Vscode-cpptools: Cpptools takes too much memory (stuck processing filename cache)

[arnonshp]

I have a very large linux project, including thousands of files.
I noticed that after some time cpptools starts consuming very large memory size and CPU consumption is very high as well. Systems becomes very sluggish.

Here's what I get when I run top:

PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 0 21.8g 11.6g 8452 S 100.7 74.1 1031:00 cpptools

I found C_Cpp.workspaceParsingPriority setting which helped in bringing the CPU consumption down.
Any special setting to limit memory consumption?

[sean-mcmanus]

How much memory is cpptools using? What OS are you using? Thousands of files should be okay -- is it hundreds of thousands of files?

This could be the same issue as https://github.com/microsoft/vscode-cpptools/issues/3123 .

[TTimo]

I have this problem as well. I set workspaceParsingPriority to lowest this morning and it has helped somewhat - the system doesn't become unresponsive as readily as before, but it still does. Especially when a build is going, which often leads to an OOM kill of the code process.

I see the cpptools process using between 4.5 and 5 GB of RAM (that's 15 to 20% RAM as reported by htop on my 24GB system).

This is on a Linux (Manjaro) machine, running KDE Plasma, core i7 CPU with 24GB RAM.
cpptools version v0.30.0-insiders3

About 60k .cpp/.h files, with a total of 160k files in the workspace folder. 420 kloc

Are there settings and commands to trace and obtain more information about what cpptools is doing and where all the cpu and memory resources are going?

[sean-mcmanus]

Yeah 4.5-5 GB for cpptools is abnormal. If you (temporarily) set your compilerPath to "" and your browse.path and includePath to [] does the memory usage go down? What causes the memory to increase? Is opening a file required?

The issue still repros when you change the C_Cpp.updateChannel to "Default", right?

We've received a bunch of reports of this in the past, but we haven't been able to obtain a repro yet. We haven't identified yet which component is using the memory.

[AnthonyParkssonos]

i had cpptools take up a whopping 15 GB of memory out of 16 GB. i had been having a million issues with vscode this week. my SSH was dropping every hour, my machine had ground to a halt. i looked at top and found that cppcode was basically destroying my machine because it can't figure out if a workspace has so many files that it would cause my machine to crash. this puts the burden on users who work with monolithic code bases. i guess i will have to really restrict the size of my workspace, but for now i have uninstalled cpptools. i have had zero issues with dropping connections or performance as a result. global search is workable instead of goto definition. the file watcher has no safety mechanism

[TTimo]

There were no settings for compilerPath, browse.path and includePath. I made sure to provide default entries in settings.json, and reverted to "Default" channel and 0.29.0 extension.

Loading the project and editing a few files does not seem to cause too much trouble for cpptools. Although at startup, even with no C++ files open in the editor, it is using 100% CPU (I watched it go for 5 minutes to see if it would taper off, and as it didn't, I started opening files to get some work done).

Then after my morning perforce get was done, I started a build, and I'm seeing this: https://gyazo.com/0f508190ba9272e2799d64609a23a9e2

watcherService is going crazy during the build. Maybe that's the root of the problem? There's a lot of .o .P etc. file activity in the tree for sure, we do a bit of on the fly .cpp generation but it's by no means huge. Does cpptool just react to watcherService spamming it?

[sean-mcmanus]

The "compilerPath", "browse.path", and "includePath" are properties in c_cpp_properties.json (or C_Cpp.default.* settings).

Yeah, we have file watchers in the workspace -- it could cause problems with causing too many IntelliSense updates, but I'm not sure how it could trigger high memory usage. We could look into that repro later. Adding the build folder to files.exclude might help.

If you set C_Cpp.loggingLevel to "Debug" you may be able to see what processing is getting triggered from the build changes.

[TTimo]

There doesn't seem to be a files.watcherInclude setting - I see little purpose for the IDE to watch anything other than my .cpp / .h files? I can't quite exclude a build folder, this build system has a tendency to put object files everywhere ..

So I added !**/*.{cpp,h} to files.watcherExclude .. I'm not sure that syntax is actually legal and supported, but I haven't noticed any new errors either. In any case, it either worked or borked the watcher service completely - I had no problems with code causing system hangs today, that's a win!

[arnonshp]

How much memory is cpptools using? What OS are you using? Thousands of files should be okay -- is it hundreds of thousands of files?

This could be the same issue as #3123 .

Sorry for the late response. As my initial post indicated, cpptools was using 11.6GB. I'm using Debian OS. My workspace is over 200k files.

[AnthonyParkssonos]

one of the problems here is that for those of us who work on monolithic codebases, it becomes inconvenient to start doing 'watcherExclude' for every folder. why not simply support 'watcherInclude' as a mutually exclusive option? or use something akin to a Ctags-like system where we build the database of references and update it on command? chokidar as it stands is similar to CLion in that for sufficiently large workspaces, watching for changes takes up all the system's resources. this makes no sense. for trivial projects, ok sure no problem, but for large problems it seems more challenging.

[sean-mcmanus]

@AnthonyParkssonos watcherExclude is a VS Code setting, so you'd have to ask VS Code for a watchInclude setting. Also, our extension may need to watch other files, such as c_cpp_properties.json if you update that. Also, the VS Code watcher code is probably not causing the memory issue -- if you think our extension has a memory leak bug when too much file watcher changes occur I could file a new bug to track that.

@arnonshp Are you able to run a memory profiler to call stacks for the allocations that are causing the memory problem? We have a wiki with some guidance on doing CPU profiling at https://github.com/microsoft/vscode-cpptools/wiki/Troubleshooting-Performance-Issues, but we haven't added info on how to do memory profiling yet.

[AnthonyParkssonos]

@sean-mcmanus does your team support for dynamic memory profiling or something akin to valgrind-massif? im assuming cpptools must be doing some kind of dynamic memory allocation?

alternatively, can Telemetry be instrumented to obtain performance data?

[sean-mcmanus]

We do dynamic memory allocation. I don't know what you mean by our team supporting memory profiling. If we can get a repro of the memory issue we could profile it to find the cause (we haven't investigated the potential file watcher repro yet).

We might be able to get a measure of the number of users impacted by excessive memory usage, but we would have to have some idea on what data structures are becoming too large in order to add telemetry on those specific cases.

[arnonshp]

@sean-mcmanus please provide memory profiling instructions and I'll run it on my system. The link you sent deals only with CPU profiling. Could not understand which perf command/args I need to use to profile memory. If there's another tool you would like me to use, please let me know.

[shammabeth]

I'm using v1.1.1 and cpptools keeps eating up all my memory (65GB+).
I do have a very large workspace with a lot of files. Regardless, it definitely feels like there needs to be a higher limit on this.

What can I provide that will help you debug this?

[AnthonyParkssonos]

the only ways i have been able to work around this is to either: 1) disable cpptools entirely 2) restrict my workspace to a subfolder in my codebase and enable cpptools for that alone, which restricts the memory usage.

[TTimo]

According to https://github.com/microsoft/vscode-cpptools/issues/6230 - IntelliSense is being killed if it goes past 4GB, starting in 1.1.0

So that's either not working, or you are seeing a different issue.

[AnthonyParkssonos]

@TTimo, Intellisense isn't just being killed, i'm seeing this:

[TTimo]

Oh you're on Windows. As I understand it's a completely different mechanism there. This discussion has been mostly about memory consumption on Linux.

[shammabeth]

I have a solid repro for this bug, to the point that it's become super annoying.
I have a terminal window ready to go always. Whenever this happens I end up killing cpptools manually and then restarting vscode. It's usually eaten up all of my 64GB RAM by that time. Sadly, turning off cpptools is not an option, as I need to use the debugger.

Is there any logging I can turn on and send?

[TTimo]

@shammabeth have you tried adding a bunch of patterns to files.watcherExclude (see https://github.com/microsoft/vscode-cpptools/issues/6035#issuecomment-683175161) - my problems went away after I did that.

[sean-mcmanus]

@shammabeth We would be interested in getting a repro for the cpptools memory problem. This might be related to https://github.com/microsoft/vscode-cpptools/issues/6424

[arnonshp]

According to #6230 - IntelliSense is being killed if it goes past 4GB, starting in 1.1.0

So that's either not working, or you are seeing a different issue.

Killing the IntelliSense when it reaches 4GB is not the solution I was expecting. It's the same as saying "we do not have IntelliSense support for large workspaces." The IntelliSense engine should be capable of crunching large workspaces without increasing memory imprint. Using disk storage along with some smart caching mechanism is just one option. Killing the engine and starting the parsing again every 4GB would make it useless for large workspaces.

[sean-mcmanus]

@arnonshp IntelliSense memory size (in cpptools-srv) is not related to workspace size. cpptools-srv only operates over a single TU (.cpp plus headers), so you can get IntelliSense to use many GB simply by preprocessing all the C++ system headers into a single .cpp file. You can change the C_Cpp.intelliSenseMemoryLimit to handle unexpected scenarios for Linux/Mac (for Windows our shipped binary is 32-bit, but we could provide a 64-bit version if there's a particular need for it).

Or are you getting cpptools using too much memory? That case might have a memory issue that scales based on the workspace size (there are some known issues). I think this issue is tracking that.

[arnonshp]

Thank you Sean. I'm sorry, I thought the cpptools large memory consumption was directly related to the IntelliSense DB which is created.
I opened this issue originally on cpptools large memory consumption (see first post on this issue).

[sean-mcmanus]

@arnonshp My best guess is that your memory issue with cpptools is related to your workspace size and the "filename cache" system. You can confirm this and get diagnostic info via setting the C_Cpp.loggingLevel to the hidden value of "7" and looking for logging that in the C/C++ pane that says "Done populating filename cache." followed by time/memory metrics (you'll probably want to change that logging level later, since it could be very verbose and reduce performance).

[arnonshp]

Do I need to restart VS code after setting the log level to "7"? I haven't done so, but did notice the following:

Populating file name cache...
fork complete (parent process, child pid = 19392)
terminating child process: 19392
Files in filename cache: 330589
Unable to determine real path of <my_path>/.<my_ws_name>.code-workspace.swp. Error = 2

There were no prints in the log thereafter.

I also noticed messages starting with "update_file_if_needed:" which point to folders which are in the filesExclude. Is that the expected behavior?

[sean-mcmanus]

Yes, you got logging level 7 -- looks like the code is "stuck" processing the 330k files in the filename cache.

@Colengms Do you have any enough info to know what is going on?

[sean-mcmanus]

If you see "update_file_if_needed" for files that are in "files.excludes" that means we're not excluding those files as expected. Make sure your files.exclude setting and C_Cpp.exclusionPolicy are correct.

[Colengms]

@arnonshp Could you connect a debugger to the cpptools process while in this stuck state, to get stacks?

[Colengms]

Hi @arnonshp . I'm looking at some code we suspect might be related to this problem, and am having trouble identifying how it could consume so much memory, even with hundreds of thousands of files. Is it possible you have a symbolic link somewhere in your workspace that is creating circular reference? If we were parsing the same files repeatedly with increasing path depth, that could lead to an issue like this.

One way you might be able to verify this would be, after repro'ing the issue, opening the tag parser database with a SQLite DB viewer, to look at the list of files it is tracking.

[arnonshp]

@sean-mcmanus

If you see "update_file_if_needed" for files that are in "files.excludes" that means we're not excluding those files as expected. Make sure your files.exclude setting and C_Cpp.exclusionPolicy are correct.

Exclusion policy is set to checkFolders and files.exclude includes the right settings. For example, in my workspace files.exclude there is an entry "**/build*x86*" and I still see messages with folders under the following path: "xyz/build-1.2.3-x86-64-xxxx"

@Colengms

Hi @arnonshp . I'm looking at some code we suspect might be related to this problem, and am having trouble identifying how it could consume so much memory, even with hundreds of thousands of files. Is it possible you have a symbolic link somewhere in your workspace that is creating circular reference? If we were parsing the same files repeatedly with increasing path depth, that could lead to an issue like this.

One way you might be able to verify this would be, after repro'ing the issue, opening the tag parser database with a SQLite DB viewer, to look at the list of files it is tracking.

I'm pretty sure I don't have circular references in my workspace. Could you please point me to the location of the tag parser DB so that I could verify this?

[sean-mcmanus]

I'm not reproing the files.exclude issue on Linux or Windows with 1.1.3. Using **/build*x86* excludes source files under "xyz/build-1.2.3-x86-64-xxxx" from being tag parsed during the workspace parsing phase and removing that files.exclude entry causes it to e added. Your scenario might be doing something special that is triggering the parsing in a way that bypasses the exclude checking.

The location of the browse database is easiest to obtain via changing the C_Cpp.default.browse.databaseFilename workspace folder setting (or the equivalent setting in c_cpp_properties.json) to "${workspaceFolder}/.vscode/vc.db" (otherwise it's gets placed in some system default location that is hard to remember). With DB Browser for SQLite, you can use the "Browse Data" tab and select the "files" table.