Vscode-cpptools: Missing include path caused cpptools-srv to grow to 5.5G. Only 400MB needed after missing path was added.

[psclkhoury]

Type: LanguageService

Intellisense is not working in a certain file. I don't know the cause yet so I am not able to make a simple repro.

I see in the logging

Failed to read response from server: 20
Resetting IntelliSense server: C:\SOMEFILE.CPP

In the extension host logging, I see

[2020-11-02 15:37:49.425] [exthost] [error] onWillSaveTextDocument-listener from extension 'ms-vscode.cpptools' threw ERROR
[2020-11-02 15:37:49.425] [exthost] [error] Error: timeout
    at Timeout._onTimeout (c:\Program Files\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:638:581)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)

But I get this message when opening most files so I dont think this is the cause.

Is there a way that I can get more info/logging on what is causing it to fail?

Extension Version 1.1.0-insiders3: October 22, 2020

Version: 1.50.1 (system setup)
Commit: d2e414d9e4239a252d1ab117bd7067f125afd80a
Date: 2020-10-13T15:06:15.712Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.17763

[sean-mcmanus]

Can you attach a debugger to cpptools-srv.exe to get a crash call stack (see https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv ). Info from C/C++: Log Diagnostics might also help.

[psclkhoury]

Here is the callstack:

KernelBase.dll!_RaiseException@16() (Unknown Source:0)
cpptools-srv.exe!__CxxThrowException@8() (Unknown Source:0)
cpptools-srv.exe!__scrt_throw_std_bad_alloc(void) (Unknown Source:0)
0388f4f0() (Unknown Source:0)
[Frames below may be incorrect and/or missing] (Unknown Source:0)
cpptools-srv.exe!edge::file_buffer::open(char const *,struct snapshot::snapshot_session_t,enum msvc::buffer_kind_e,struct a_canonicalized_file_name *) (Unknown Source:0)
cpptools-srv.exe!edge::file_buffer::open(char const *,enum msvc::buffer_kind_e) (Unknown Source:0)
cpptools-srv.exe!a_file_buffer_stream::create(char const *) (Unknown Source:0)
cpptools-srv.exe!open_input_stream(char const *) (Unknown Source:0)
cpptools-srv.exe!open_input_stream_with_result(char const *,char const *,struct an_open_file_result *) (Unknown Source:0)
cpptools-srv.exe!open_source_file(char const *,struct an_open_file_result *,enum a_unicode_source_kind_tag *) (Unknown Source:0)
cpptools-srv.exe!open_source_file_with_error_handling(char const *,int,struct an_open_file_result *,enum a_unicode_source_kind_tag *) (Unknown Source:0)
cpptools-srv.exe!try_to_open_source_file_if_not_already_included() (Unknown Source:0)
cpptools-srv.exe!search_for_input_file() (Unknown Source:0)
cpptools-srv.exe!open_file_for_input_full(char const *,int,int,int,int,int,int,int,int,int,char const *,char const * *,char const * *,char const * *,struct an_input_stream * *,int *,enum a_unicode_source_kind_tag *,struct a_directory_name_entry * *) (Unknown Source:0)
cpptools-srv.exe!open_file_and_push_input_stack(char const *,int,int,int,int,int,int,int,int,int *,int,int,int,char const *,char const * *) (Unknown Source:0)
cpptools-srv.exe!proc_include() (Unknown Source:0)
cpptools-srv.exe!pp_directive(void) (Unknown Source:0)
cpptools-srv.exe!get_token(void) (Unknown Source:0)
cpptools-srv.exe!scan_nonmember_declaration(struct a_decl_parse_state *,struct a_source_range *) (Unknown Source:0)

I also saw this in the debug console, looks related:

WER/CrashAPI:2658: ERROR Invalid args, too  big block

[sean-mcmanus]

Thanks for the call stack. We've been able to locate the line of code that is crashing.

[sean-mcmanus]

Actually, it looks like we identified a different null reference but this is a bad_alloc, which I don't see the cause for. Do you have any more repro details in regards to what is special about that particular file, such as any particular file encoding?

[psclkhoury]

@sean-mcmanus Most of the files are ASCII text, with CRLF line terminators and ANSI (Windows-1252) encoding on Windows.

[Colengms]

Hi @psclkhoury A bad_alloc exception would seem to suggest running out of memory. cpptools-srv.exe is currently a 32-bit process on Windows. Do you see it grow to ~4G as this occurs?

[psclkhoury]

@Colengms After updating to Version 1.1.0-insiders4: November 3, 2020 of the extension, I started having the same issue in another file and not in the previous file anymore.

I get the exception at around 3GB:

[Colengms]

I believe you are seeing an out of memory issue. Is the code involved extremely complex, such as a template library? Or, it's possible you are repro'ing a bug that is causing memory to grow when it should not. Either of these possibilities would relate to the code being parsed, and investigating it would require an isolated repro. Is this code you would be willing to share, so we can investigate the repro?

[sean-mcmanus]

@Colengms Maybe we could release the 64-bit Windows offline vsix for 1.1.0 for users to try if they require more memory on Windows -- that would allow us to determine if the memory usage is just "high" (i.e. "by design" with possible room for optimization) or "unbounded" (i.e. a bug).

[psclkhoury]

@Colengms It's not very heavy in templates, but it has a lot of includes and there is a lot of files in the repo. Unfortunately I am not allowed to share the code but I will try to find a repro.

[sean-mcmanus]

This is the 1.1.0 64-bit Windows VSIX: https://download.visualstudio.microsoft.com/download/pr/17062e60-bd73-48b7-bc31-54d71ef5d48d/2895a0d0eda48a734f6575e103524c42/cpptools-win64.vsix .

If you can try that version and see what memory usage occurs that may solve your issue (or use unbounded memory). We haven't released that vsix more generally because we're not sure yet how useful it would be compared to the 32-bit version and it has the potential of using more memory than desired.

If that works, you'll probably want to increase the value of C_Cpp.intelliSenseMemoryLimit (or else the IntelliSense process will shutdown repeatedly when memory usage is over the limit).

[psclkhoury]

After trying the 64bit version, I think I know what is causing the bad alloc exception.

When I used the 64bit version, the used memory kept increasing until it reached ~5.5GB. After that, there was a logging that the extension will fallback to the tag parser for intellisense operations, and there was an include error in the problems panel.

When I fixed the missing include path by adding it to the c_cpp_properties.json, intellisense started to work again and memory consumption was 'only' at 400MB.

I have a lot of include paths in c_cpp_properties.json (~3800 paths), so maybe there is a memory leak in cpptools-srv when trying to resolve the includes, and the big number of include paths makes it more apparent?

[Colengms]

Hi @psclkhoury . To confirm, the issue with memory use was cleared up by adding an include path that was missing, and memory usage has been reduced to 400MB despite ~3800 include paths still in use?

This seems like either a leak related to resolving headers, or perhaps due to code that IntelliSense continued to parse after not finding that header, which it was possibly parsing incorrectly due to missing information. We'll use this issue to track further investigation.

If you are able to reduce the repro to a self-contained example that we could use to investigate, that could help, but I understand if that's not feasible.

[psclkhoury]

Hi @psclkhoury . To confirm, the issue with memory use was cleared up by adding an include path that was missing, and memory usage has been reduced to 400MB despite ~3800 include paths still in use?

@Colengms Yes that is correct.

If you are able to reduce the repro to a self-contained example that we could use to investigate, that could help, but I understand if that's not feasible.

I will try to come up with a self-contained example. I will post it here if I manage to do it.

[Colengms]

We think we've identified what's happening. It appears to be related to a cache of locations tried while searching for a header file, that can grow excessively. We have a way to greatly reduce the memory usage. We'll use this issue to track that work.

[sean-mcmanus]

Also, FYI using 3800 includePaths will probably negatively impact performance of cpptools-srv.exe. Using recursive includes (paths that end with "**") might improve performance via eliminating include paths that are not actually used in a TU.

[psclkhoury]

@sean-mcmanus Thanks for the tip, but I already tried to put only the root folder of all the includes appended with "**" but it doesn't work. I will get suggestions to add missing include paths to c_cpp_properties.json and the tag parser will be used instead of intellisense. Even though all of the includes are under ${workspaceFolder} or the other root path.

The simplest way I found is to parse the include paths used by our build system and add all the paths that are in the repo to c_cpp_properties.json.