Sdk: dotnet build leaves running dotnet.exe processes after it finishes (.csproj specific)

[jhudsoncedaron]

Steps to reproduce

.csproj files attached
buildleak.zip
Enter c directory
Open task manager/details view
observe no dotnet.exe processes are running (clean test environment)
run dotent build

Expected behavior

No dotnet.exe processes left

Actual behavior

dotnet.exe processes left

Environment data

dotnet --info output:

.NET Core SDK (reflecting any global.json):
Version: 2.1.300
Commit: adab45bf0c

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17134
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.300\

Host (useful for support):
Version: 2.1.0
Commit: caa7b7e2ba

.NET Core SDKs installed:
1.0.0 [C:\Program Files\dotnet\sdk]
1.0.1 [C:\Program Files\dotnet\sdk]
1.0.4 [C:\Program Files\dotnet\sdk]
1.1.0 [C:\Program Files\dotnet\sdk]
2.0.0 [C:\Program Files\dotnet\sdk]
2.1.200 [C:\Program Files\dotnet\sdk]
2.1.300 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[livarcocc]

This is by design in 2.1.300.

As part of our perf improvements, we introduced three persistent servers with the goal of reduce our JIT time. The servers are the razor compilation server, the vbcscompiler server and the msbuild node re-use server.

If you don't want them staying around when you are done building, you can invoke dotnet build-server shutdown to turn them off.

If you don't want them to start to begin with, you can set different properties/environment variables to prevent them from starting. From @peterhuene:

Use -p:UseRazorBuildServer=false to disable the Razor (rzc) server.

Use -p:UseSharedCompilation=false to disable the Roslyn (vbcscompiler) server.

For MSBuild, pass /nodeReuse:false on the command line to disable node re-use.

[jhudsoncedaron]

They're causing spurious build failures due to holding locks.

[eikc]

@livarcocc I have a whole bunch of teamcity agents that are having the same issue as @jhudsoncedaron. The processes somehow doesn't release the files under a build of a project. random json files/dll's are not released and the build fails..

If i try to rerun the same build, it fails again. If i kill the 3 .NET core host processes, i can successfully build again, once or twice then the error occurs again.

Build steps are like:

1. dotnet restore
2. dotnet build
3. dotnet test
4. dotnet publish

It can fail randomly under each step.

This happened after we upgraded our TC agents with the latest version of dotnet core:

.NET Core SDK (reflecting any global.json):
Version: 2.1.302
Commit: 9048955601

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.302\

Host (useful for support):
Version: 2.1.2
Commit: 811c3ce6c0

.NET Core SDKs installed:
2.1.4 [C:\Program Files\dotnet\sdk]
2.1.105 [C:\Program Files\dotnet\sdk]
2.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[livarcocc]

Have you tried the steps that I suggested above?

[eikc]

Yes, i have tried all of the above and the builds continued to fail

I ended up adding /m:1 as i read on stackoverflow to disable the multi-process build and now every build is green.

I'm not sure if my issue is related to dotnet/cli#9648

[jhudsoncedaron]

But that's (#9648) my issue. If it weren't for the fact that this is the older number I'd say it was a duplicate.

[jasondaicoder]

dotnet build-server shutdown doesn't work on mac, I can still see the dotnet processes are live and consuming lots of CPU.

[peterhuene]

@jasondaicoder what DLLs are those dotnet processes executing? That is to say, if you view the command lines for those processes, what was the argument to dotnet or dotnet exec?

[TheXenocide]

We are also experiencing locked files in build environments being held by the dotnet process. If the work it's performing is done, shouldn't it be releasing the files it was using? Even in the case of Razor, different builds could use different versions of the package so shouldn't there be some sort of AssemblyLoadContext to allow GC or a non-locking load behavior (load by byte-array, copy before load, etc.)? This also prevents clearing the NuGet package cache, even in idle environments.

EDIT: here's an issue from TeamCity restoring packages

[18:46:52]
[restore] Access to the path 'Microsoft.AspNetCore.Razor.Language.dll' is denied.
[18:46:52]
[restore] Process exited with code 1
[18:46:52]
[restore] Process exited with code 1 (Step: Restore NuGet Packages (NuGet Installer))

[abeham]

Same here, linux gitlab build server had lots of dotnet processes running /usr/share/dotnet/dotnet /usr/share/dotnet/sdk/3.1.101/MSBuild.dll /usr/share/dotnet/sdk/3.1.101/MSBuild.dll /nologo /nodemode:1 /nodeReuse:true from different dates. I counted at least 23 instances. Build hang in restore. I killed all the processes now it works, but for how long? Should there really be 23 instances all running under the same user?

[jhudsoncedaron]

So in the years, most of the "locks" problems have been fixed, but there are two big problems left.

1) There's one process or set of processes started per directory you build from.
2) Under Windows, a process holds a lock on the directory it was started from, even if its current directory is changed later. In order to get rid of this lock pileup, the spawning process needs to carefully change its current directory to C:Windows or something like that before starting the persistent child.

[jjxtra]

I have dozens of these lingering processes, please revert the default behavior to not leave them hanging around