Fsharp: Visual Studio 2017 hangs, crashes when opening large F# solution

[asik]

We have a large (~90 projects) solution, mostly F#. In VS2015 it takes maybe 20-30 seconds to open and the memory usage is around 400MB. In VS2017, the IDE hangs after loading all the projects, memory usage quickly climbs in the 3GB range, and it can stay like that for several minutes before eventually crashing.

This seems related to F# as larger C# solutions don't present this problem.

This is closed-source so of course I cannot share the code, but I would be glad to provide any non-sensitive data to help diagnose this issue. I tried simply creating a new solution and adding our projects to it one by one. Around 60 projects the IDE became extremely slow and usually cannot open the solution anymore. If it manages to get past this loading phase, memory usage goes back to something normal. Then sometimes it becomes usable, sometimes it crashes anyway.

I just tried this again with VS Community 2017 (15.0.0+26228.12) and Visual F# Tools 15.4.1.17040401.

Here's a vmmap screenshot while it's hovering around 3GB:

[vasily-kirichenko]

Would you mind to take a gotMemory snapshot and share it?

[vasily-kirichenko]

Do I understand correctly that no files are open?

[asik]

Yes I just open Visual Studio 2017 and open the solution.

Not sure about sharing a dump, it's going to have file names and whatnot... but here are some screenshots in dotMemory when devenv.exe was around 1GB and climbing:

Maybe 2 million FSharpChecker objects is a lot? Or maybe going by the vmmap info it might not be managed objects at all?

[vasily-kirichenko]

Wow o_O. There should be strictly one FSharpChecker

[vasily-kirichenko]

Can you show retaining paths for FSharpChecker?

[asik]

Oh wait, the number on the right is the number of objects _under_ it. So there are only 2 FsharpChecker objects here (the number on the left).

[asik]

As vmmap suggests it looks like the growth in .NET objects is not the problem, it's mainly unmanaged memory:

FWIW though here are the deltas for .NET objects, sorted by Objects delta:

[vasily-kirichenko]

From how many projects do you have files opened? Can you open a single file from any project (and restart)?

[asik]

So this is all happening before any files are open in the editor. VS is hung on the Start Page, the Solution Explorer is empty. If it manages to load it (I have a smaller one with 61 projects that just barely loads, sometimes), right now one .fs file opens in the editor. Then it's kinda laggy and sometimes it manages to fall back on its feet and become usable, sometimes it crashes anyway. The diff from my last post was on this solution.

The full 90 projects solution never makes it to that stage though. You get the dialog saying it's loading the projects, that dialog completes and disappears, then it just stays there and eventually crashes.

[vasily-kirichenko]

Do you mind try https://drive.google.com/open?id=0B8HLQUKik9VtNFF1UEZNaElhZTg? I reduce project cache size from 200 to 5 in this build. First, open the dev command prompt and run

Common7\IDE\VSIXInstaller.exe /uninstall:VisualFSharp

Then run my vsix.

[asik]

Using your VSIX, I get "The 'FSharpProjectPackage' package did not load correctly in VS". In ActivityLog.xml this seems relevant:

System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Users\***\AppData\Local\Microsoft\VisualStudio\15.0_4ee222e8\Extensions\thoyrmux.2tv\FSharp.ProjectSystem.FSharp.dll' or one of its dependencies. Strong name validation failed. (Exception from HRESULT: 0x8013141A)
File name: 'file:///C:\Users\***\AppData\Local\Microsoft\VisualStudio\15.0_4ee222e8\Extensions\thoyrmux.2tv\FSharp.ProjectSystem.FSharp.dll' ---> System.Security.SecurityException: Strong name validation failed. (Exception from HRESULT: 0x8013141A)

[vasily-kirichenko]

Ah, you have to disable strong validation checking.

Another thought: can it be the case that your solution has circular project references?

[vasily-kirichenko]

sn -Vr *,*

[asik]

The solution doesn't have circular references. I thought this was impossible but I was curious so I wrote a program to verify it; it really doesn't.

I'm seeing the same behavior using your VSIX.

[asik]

Well, on the smaller (61 projects) solution, with your VSIX it seems to more reliably open it and let me use it, after a very long loading time and going well into 2GB of RAM. But the full one still crashes as before.

[vasily-kirichenko]

@majocha can it be the case that your "register all projects in workspace as solution is loading" feature can eats all memory and take very long time to finnish?

@asik can you take a dotTrace "timeline" snapshot while it's loading your full solution?

[majocha]

@vasily-kirichenko I think it did the same before, just not on load but after opening first file.

[vasily-kirichenko]

@majocha no. It loads the project a file belongs to and all referenced projects.

[saul]

@vasily-kirichenko that's my concern too. I noticed that there's a call in ProjectSiteAndFiles that is O(2^n) where n is the number of projects in the Solution (with a worst case dependency graph).

It's high on my list to investigate, but I imagine that memoising that function will significantly improve load time performance.

My other concern is that Roslyn workspace setup is done async but it context switches back to the UI thread, locking everything up 😔

[majocha]

@vasily-kirichenko that was the case for some time but before my change that was a PR that queued all the projects and processed them at once after file open. So it's not entirely my fault :)

[saul]

The O(2^n) function I'm referring to is ProjectSitesAndFiles.getProjectOptionsForProjectSite

[vasily-kirichenko]

I cannot reproduce it - all my solutions load quite fast. @asik a dotTrace snapshot would be very much appreciated.

[saul]

@vasily-kirichenko, this should help you reproduce the issue that @asik is seeing:

1. Download and extract Perf.zip
2. Run gen.fsx
3. Open dense/Dense.sln in Visual Studio

[vasily-kirichenko]

@cloudRoutine this thing should be cached https://github.com/Microsoft/visualfsharp/blob/master/vsintegration/src/FSharp.LanguageService/ProjectSitesAndFiles.fs#L111, like

static let solutionService (serviceProvider: System.IServiceProvider) =
    lazy (try Some (serviceProvider.GetService(typeof<SVsSolution>) :?> IVsSolution) with _ -> None)

[cloudRoutine]

why can't we just pass it the IVsSolution as an arg?

[Pilchie]

Just wanted to point out that the call to serviceProvider.GetService() should effectively be a dictionary lookup in the shell after the first call, so caching it locally in F# isn't likely to help much.

[asik]

@saul it's really cool you wrote that. It does repro the long loading time, however not the memory usage going out of control. But hopefully fixing the former would fix the latter.

I looked at dotTrace and unfortunately it also has file names of all the projects and dlls, can't share that. But if it's certain ETW events you want I can run perfview, any particular command line?

[forki]

First PR addressing part of the problem is at https://github.com/Microsoft/visualfsharp/pull/2802

[dsyme]

@asik It would be really, really great if you could manufacture an entirely artificial huge solution that exhibits this problem. I know it might take a bit of work generating fake projects and code, but if we can capture the solution then we can put it in our automated test cases and ensure that we always scale to a solution of this size. This would be a hugely appreciated contribution if you could manage it.

[asik]

Good news! I can actually repro with @saul 's generated solution, on my work environment. I have no idea why in this environment the memory balloons out of control, whereas at home it doesn't. But maybe at home I'm not running the latest nightly. I'll retest when I get time.

So anyway, dotTrace snapshot from VS slowly dying:
https://www.dropbox.com/s/13swowphpptk8a4/VS2017%20timeline%20snapshot.7z?dl=0

[vasily-kirichenko]

@asik could you try https://github.com/Microsoft/visualfsharp/pull/2801, grab vsix here https://drive.google.com/open?id=0B8HLQUKik9VtSlBtVTdfUHhYMjQ?

[asik]

So it does load our solution, pretty fast as well. However, after that, trying to open files from the solution explorer I am often greeted with

and there is no syntax highlighting at all, even on small solutions. Just walls of white text.

[saul]

@asik that is left over from Vasily's investigation. To fix it, create that file on your disk.

[vasily-kirichenko]

That VSIX is not usable anyway, it's just POC where the problem is. I'm not gonna provide a fix.

[dsyme]

@saul @vasily-kirichenko @cloudRoutine I added the stress test projects generated by @saul directly to the repo in https://github.com/Microsoft/visualfsharp/pull/2997. They are in tests\projects\stress

Checking in the files directly is a bit brutal but we need to have at least a few project stress tests we can share, focus on and automate

[dsyme]

Perf info for opening tests\projects\stress\medium\dense\Dense.sln:

Inclusive:

Exclusive:

[dsyme]

I believe I know how to fix this problem. We will have to change the shape of CompileOptions to no longer recursively specify the dependent projects but rather provide a list of projects options indexed by DLL name. I will work on a PR for this tomorrow

[dsyme]

@saul I've merged #3002 and will close this bug to match, but please submit your run-on-the-background improvements too