Roslyn: VBCSCompiler.exe does not end when building .Net Core 3.1 app from Visual Studio 16.4.5

[jagbarcelo]

Version Used:
We've just updated to Visual Studio 16.4.5 a couple of days ago, but this issue has been hitting us for quite some time (sure since 16.4.3 and maybe even before that). Windows 10 1909 (Build 18363.628).
VBCSCompiler.exe 3.400.119.61401

Steps to Reproduce:
We have a solution with 19 projects (most of them libraries). We are building a client/server solution so we have a Web App (running .Net Core 3.1, exposing a RESTful API) and a Client, which is a WPF app. We are using the MVVM pattern, where the View is the Client WPF project, and the ViewModel and Model are a couple of libraries (DLLs)

1. We open VS2019 with this solution, and modify something so that the solution needs to be re-compiled.

2. We run the solution in debug mode for the first time (it is set up with several start projects: both the IIS Express server and the client app). Visual Studio 2019 compiles it fine this time, but it leaves VBCSCompiler.exe running in the background (zombie?). The debug sesion runs fine. We exit the client app and close the browser (which was opened automatically by VS because of the web app is also launched).

3. We make further changes in the code and try to compile/debug again.

4. This second time, the compilation fails with errors MSB3027 and MSB3021:

MSB3027 Could not copy "G:\<my-solution>\Library.ViewModel\bin\Debug\netcoreapp3.1\Library.ViewModel.dll" in "bin\Debug\netcoreapp3.1\Library.ViewModel.dll". Exceeded retry count of 10. Failed. The file is locked by: "Microsoft Visual Studio 2019 (65232)"
MSB3021 Could not copy "G:\<my-solution>\Library.ViewModel\bin\Debug\netcoreapp3.1\Library.ViewModel.dll" in "bin\Debug\netcoreapp3.1\Library.ViewModel.dll". The process cannot access the file 'bin\Debug\netcoreapp3.1\Library.ViewModel.dll' because it is being used by another process.

1. We guess that the proceess VBCSCompiler.exe which is left running from the 1st compilation is the culprit, but as this lock happens most of the times (not always), there might be something else involved that we do not yet know.

2. a. When the errors appear, manually killing VBCSCompiler.exe does not help (VS still cannot compile, with same errors). We need to exit Visual Studio completely and restart it in order to compile a second time.

   b. If instead of killing VBCSCompiler.exe and exiting VS, we just exit VS, a couple of processes started by VS are left orphaned and running:

   • One is VBCSCompiler.exe, started with the following command line: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\VBCSCompiler.exe" "-pipename:cg5s3tyW_oNuSv9ytFEzucuhDSrlb7x2CDdEq6ZbIrc"
     This is confirmed with Process Explorer, which reports Parent: <Non-existing Process> (instead of MSBuild.exe, which is shown when VS is still running).
   • The other one is dotnet.exe, also initiated by VS with: "dotnet" "C:\Program Files\dotnet\sdk\3.1.101\Sdks\Microsoft.NET.Sdk.Razor\tools\netcoreapp3.0\rzc.dll" server -p MYUSERNAME.4htt1K2+mTbFo21UFxvtMknhg
     Process Explorer also report Parent: <Non-existing Process> when left running (instead of MSBuild.exe).

   c. Sometimes, when compilation errors MSB3027 and MSB3021 do not appear, exiting VS still leaves those two processes around. So, there must be something wrong around anyway despite the fact that it does not show up with errors when compiling.

We are running the Spanish-localised version of Visual Studio 2019 (and a Windows 10 in Spanish) but I do not think that has anything to do with the described problem.

Expected Behavior:
Visual Studio should compile not only the first time, but any subsequent times without the need to close it and start it again.
As you can imagine, developing this way has turned into a nightmare.

Actual Behavior:
We believe this is something similar to #37456 (or another regression).

Since this is a solution quite large, making a simpler repro would be desirable, but we do not know where to start trimming the branches. Any help would be appreciated. We could also provide a dump or mini dump of VBCSCompiler.exe if required.

[jcouv]

• We guess that the proceess VBCSCompiler.exe
• manually killing VBCSCompiler.exe does not help

This makes me think that this was not the guilty process.

Could you positively identify the process that holds the lock? There's a few methods you can use (here's a few).

Visual Studio 2019 compiles it fine this time, but it leaves VBCSCompiler.exe running in the background
We believe this is something similar to #37456 (or another regression).

It is expected that VBCSCompiler.exe would continue running. This saves us from loading referenced metadata files multiple time. The fact that your first compilation actually completes and succeeds tells me this is not similar to #37456.

[jagbarcelo]

@jcouv

This makes me think that this was not the guilty process.

I think so too, but what about the fact that it is left running and orphan when VS closes? And the other process dotnet.exe too.

Could you positively identify the process that holds the lock? There's a few methods you can use (here's a few).

I'll have a look at it next monday. We really need to get to the bottom of this because it is starting to impact our development seriously.

It is expected that VBCSCompiler.exe would continue running. This saves us from loading referenced metadata files multiple time. The fact that your first compilation actually completes and succeeds tells me this is not similar to #37456.

Again, it shouldn't be left around running when VS is closed. Should it?

[jagbarcelo]

@jcouv, using Process Explorer we have found that the process that is locking the DLLs compiled in the previous run is devenv.exe, i.e. the main "Visual Studio 2019" process. Thus, only a full close and re-open of VS solves the issue.

[jcouv]

Tagging @jaredpar @jinujoseph to help route this issue to VS team.

[jaredpar]

Again, it shouldn't be left around running when VS is closed. Should it?

This is expected. The VBCSCompiler process is actually controlled by MSBuild, not Visual Studio. It often appears it's controlled by Visual Studio though because Visual Studio kicks off MSBuild and that in turn will initiate VCBSCompiler.

In any case though it's the expectation that this will remain around after a build completes for several minutes. The default keep alive time is ~5 minutes after the last time that a compilation request is received.

The reason for this, as @jcouv, indicated is to speed up compilations. There are a number of data structures and caches that can be persisted between compilations. The server process stays alive after VS and MSBuild complete because the expectation is that a new compilation request will come in shortly that can take advantage of this caches. After a few minutes of no activity though the server will shut itself down and a new one will spin up on demand for the next compilation.

We guess that the proceess VBCSCompiler.exe
manually killing VBCSCompiler.exe does not help

This makes me think that this was not the guilty process.

Agree. There are two pieces of data that indicate this is not VBCSCompiler holding the files in memory:

1. Killing the process has no effect on the locked DLLs. If VBCSCompiler was locking the DLLs then killing the process would unlock them
2. The error message indicates it's Visual Studio locking the file, not VBCSCompiler: The file is locked by: "Microsoft Visual Studio 2019 (65232)

Tagging @jaredpar @jinujoseph to help route this issue to VS team.

@agocke what is the typical process for moving GitHub bugs back to VS? Do we close this and have the customer file a VS feedback issue?

[agocke]

what is the typical process for moving GitHub bugs back to VS?

If there was a VS Feedback issue open to start with, we could re-open it. Otherwise, we'll just have to close this and the customer will need to file a new issue.

[jagbarcelo]

More feedback in this issue. We've narrowed it down to this feature: Enable UI Debugging Tools for XAML which is under VS Options -> Debugging -> General.

Switching off this option (and restarting VS for the change to take effect) and the issue goes away. We've been working for two days so far without this problem.

However, when we enable this option (and if during the debugging session we change focus to VS so that Live Visual Tree is shown along with the XAML source file for the current screen in the app), then VS opens our library DLLs and leaves them opened after the debugging has finished, thus preventing further compilations afterwards (that need to replace those very same DLLs).

We are working on a simpler proof-of-concept sample (instead of our original huge solution), but so far we've not been able to reproduce the problem with simpler solutions.

It's ok if you close the issue here. I will try to open a VS feedback issue later, if we finally make a repro in a simpler VS solution.

Thanks.

[mgoertz-msft]

@jagbarcelo Thank you for narrowing this down. I have given the owners of the XAML UI Debugging Tools a heads-up of the incoming VS Feedback issue for this problem.

[jcouv]

Closed as external. Feel free to share the feedback link here for others too.

[etvorun]

We've narrowed it down to this feature: Enable UI Debugging Tools for XAML which is under VS Options -> Debugging -> General.

@jagbarcelo I am one of UI Debugging developers. Thanks for reporting this issue. Can you please share more data. Do any of your libraries have satellite designer DLLs, e.g. MyLib.dll and MyLib.Design.dll? Does your issue repro if you enable UI Debugging but uncheck Enable Just My Xaml? If you cannot create a repro project can you get VS into a bad state, then start another VS, attach to devenv.exe for managed debugging and copy content of Debug > Windows > Modules window. VS can spin off additional app domains and I would be interested to know in what app domain your dlls are loaded.

[jagbarcelo]

@GitHubUser30

Do any of your libraries have satellite designer DLLs, e.g. MyLib.dll and MyLib.Design.dll?

No, no satellite designer DLLs at all. The are just DLLs for Resources, Model and ViewModel.

Does your issue repro if you enable UI Debugging but uncheck Enable Just My Xaml?

Yes. With Enable UI Debugging Tools for XAML enabled, and all sub-options too except Just My XAML (which was disabled), the problem also shows up.

If you cannot create a repro project can you get VS into a bad state, then start another VS, attach to devenv.exe for managed debugging and copy content of Debug > Windows > Modules window

We are working on a simpler repro, but so far we have had no luck. In the meantime, after associating another instance of VS to the faulty instance, the list of modules shown in the debug window you mentioned are attached in the following CSV file (zipped):
VS-Modules.zip

I'll let you know if we manage to create a simple repro solution/project.

[etvorun]

Thanks for the info. It ruled out some possibilities. Can you please use Assembly Load Debugger (https://marketplace.visualstudio.com/items?itemName=MadsKristensen.AssemblyLoadDebugger) to get a call stack for code which loads for your assemblies.

[jagbarcelo]

@GitHubUser30 Sorry for the delay. Please have a look at the attached output files generated by the tool you mentioned. I hope it helps in any way.

AssemblyLoadDebugger.zip

[etvorun]

That pinpointed the root problem! Thanks:)

[mario-rosales]

What ended up being the solution for this?

[etvorun]

Issue have been fixed in Visual Studio 16.6.

[jagbarcelo]

We are running Microsoft Visual Studio Community 2019 v.16.7.4 and the issue is still there. We still have this issue from time to time. Yesterday once, without going much further back. I'm sorry to say it is not fixed at all.

BTW: We were not using Enable UI Debugging Tools for XAML (it was disabled) and the issue still hapened, so maybe it was not because of this option after all (or at least not the only one to blame).

[spadapet]

@jagbarcelo If this is the same problem as before where devenv.exe is locking your DLL, can you run the Assembly Load Debugger tool again (see the link in an earlier comment). Last time that gave us a callstack that loaded your DLL into devenv.exe and that one should've been fixed. Maybe there is another way your DLL gets loaded. Let's see if Assembly Load Debugger gives any info.
Thanks!

[jagbarcelo]

First of all, @spadapet, thanks for your interest.

We've just done the following:

1. Opened VS without code.
2. Enabled Assembly Load Debugger and starting a capture with it.
3. Load our solution and open some of the .XAML files so that they are rendered in screen.
4. Compile the solution in Debug mode and run it.
5. Had the application running for a while, checking some of the things it does.
6. Stop the debugging processes.
7. Changed the mode to Release.
8. Recompile again (in Release).

This last time the compiler started compiling as usual but, after some seconds, it simply hang, doing nothing, without any output to the compilation log (which is usually receiving continuous output and scrolling down as the compilation goes on). The progress bar overlay in VS taskbar icon stopped too.

After more than 15 minutes without any progress we canceled the compilation, which usually takes a max. of 3 minutes. The last message shown in the compilation log was a success compilation of the 7th project after 4.6seconds. Nothing else was logged there for 15 minutes.

After having to force the compilation to a stop, we checked the output in the Assembly Load Debugger, but there were no ocurrences of any of our libraries (most of them were System..., Microsoft..., Xamarin..., etc).

It seems this time the issue is not exactly the same as before, since there is no final error message saying that a particular file could not be copied to the output directory because it was locked by another process.

We have done more checks around this issue: If you open VS, force a full recompilation in Debug, then change to Release and do a whole recompile it again, works fine. You can change back and forth from Debug and Release and do full recompilations and everything works. However, if we do a run on the Debug version, and then try to change it to Release, this time the compilation hangs. We need to set the solution to Release, save it, close VS and reopen the solution again to be able to successfully compile it.

Summing up, we need to change the target of the compilation, save the solution, close VS and reopen it again in order to successfully compile in Release mode. Luckily, changing from Debug to Release (and back) is not something we do on a daily basis, but it is still annoying.

Any ideas what this might be? Have you seen some behaviour like this before? I can send you all the output files of the Assembly Load Debugger but I do not think you could find something there to pull from.

Thanks.

[spadapet]

@jagbarcelo

However, if we do a run on the Debug version, and then try to change it to Release, this time the compilation hangs.

When you do this, are any XAML files open in the XAML designer? Basically is step 3 in your repro steps necessary. I'm guessing you did open a XAML file, but just checking to be really sure.

There are some things you can do:

1. If the hang only happens when the XAML designer is loaded in VS, then use the "Send Feedback" button in VS's title bar so that someone on the XAML team can look at it. (that includes me, but it would be better for us to have official feedback to keep track of the investigation instead of this closed github issue).

2. Figure out which processes are hanging (VBCSCompiler.exe, msbuild.exe, etc.) and attach VS as a debugger to it. You can save a heap dump and upload the file to a VS feedback item so we can see what the threads are doing.

Without a test solution for me to repro the problem, I'm not sure what else to look at. It seems like the original fix I made did do something since your built DLLs aren't being locked by VS anymore, but something else is still going on.

[jagbarcelo]

@spadapet

I have tested the same steps as yesterday but without having files of any kind opened in the solution when both compilations were made. The hang still occurs if you run a debug session between the Debug compilation and the final attempted Release compilation (which fails).

So, it seems the culprilt of this is not the XAML designer, which shouldn't have been triggered at all during these steps. I'll try to pinpoint which process(es) is(are) hanging but so far it has been quite difficult.

For starters, I have tried to reduce the number of concurrent compiling processes down to 1 (Options -> Project & Solutions -> Compilation & Execution -> Max. number of parallel project compilations), but still, 4 - 6 different MSBuild.exe processes are executed and I cannot know for sure in which one the hang is taking place, or if it is in any of those at all! It could also be in VBCSCompiler.exe, dotnet.exe, or ServiceHub.TestWindowStoreHost.exe, or in any of the 7 launched ServiceHub.Host.CLR.x86.exe that also show activity during the stuck compilation.

I have also increased the verbosity of the logs with no luck either. At some point something simply stops and there is no output that could be used to know where to look to. So I took another approach: I started recompiling each project, one by one, in the order that the VS should be compiling them (according to their dependencies).

The first one to compile was a simple project with just common resources (mostly strings) with no issues at all.
The second one was the Model (we are using a Model–view–viewmodel pattern) and bingo... the output log showed the first compilation ended just fine (successfull compilation of the Resources with an elapsed time of 00:00:00.94) but the overlay progress bar at the VS icon in the taskbar stuck at 50%. The Compile menu still showed all options greyed out except the option to Cancel the compilation. Since we are recompiling, and the Model project has a dependency on the Resources project, it compiles the Resources first and tries to compile the Model as the second step (which does not even start, at least nothing is shown in the logs).

This Model project of our solution only has references to the former Resources project (which compiled just fine) and to the assemblies: PresentationCore, PresentationFramework, System.Drawing and Windows.Base.

Furthermore: Once VS is in this weird state, Recompiling this single Model project shows the same full output in the log console telling that the Resources compilation finished successfully, but that's all, it hangs there. Compiling without having changed anything in the project, shows no output at all and simply hangs from the start (needs to be canceled too).

The target output directories for this Model project are not locked by any process: I can simply delete them using Windows Explorer without any issue. I can even delete the whole [bin|obj]/Release/netcoreapp3.1 output directories.

Making a simpler Repro-Solution would be ideal but since the issue only happens when you RUN a debug session between a Debug compilation and a Release compilation, this Repro-Solution wouldn't be as simple as just our Resources and the Model projects, since with those only... there is not much to debug at all. I am a bit entangled here, on one hand willing to help you with this issue, on the other hand with not much time to do many tests to solve it when it can simply be avoided by closing and reopening VS.

As you can understand, I cannot open an official feedback through the builtin VS feature "Send feedback" without knowing more details or I will simply get ignored. Please bear with me, just a little more. Any more ideas where to look for to pin-point this issue a bit more?

Thanks.

[spadapet]

@jagbarcelo Right, if you open VS feedback, someone may just ask for more info and a sample project anyway. Or they might know more than I do about debugging build issues.

I'm just about out of ideas, but if I was connecting to your machine to try and debug this, the first things I'd try are:

• Use Process Explorer to search for any process that has any of your DLLs open. Like if your root project directory was called "MyProject", then Ctrl-F in procexp.exe and search for "MyProject". Sounds like you did something like this before.

• Try a command line build instead of a VS build when VS would hang. Like: msbuild.exe Model.csproj /p:Configuration=Release

  • If the command line build hangs, then there are less processes to look at

• Instead of running a debug session between Debug/Release configs, just run the app without the debugger attached. See if that still triggers a hang.

Basically I'd just keep looking for more info like you're already doing. At least there is a workaround, but it would be interesting to eventually know what's going on.