Vscode-cpptools: Suggestion: Improvements of parsing large projects

[xueyunquan]

Hi,

I'm trying to work on some large projects with VS Code and this CPP plugin. For a large project, say Linux kernel for an example, it takes a long time to parse the code. To improve the experience, I'd like to suggest the following ideas:

1. Add a parsing progress indicator on the status bar. It would be sufficient to just show a percentage of the parsing progress. Without this, it frequently makes me wonder if the parsing has completed or in progress or even hang.

2. Add a "Force Re-parsing" command, so that I can force re-parsing one, several or all files. This is required because sometimes it takes less time to manually trigger the parsing other than waiting for VS Code to detect changes in a large project, especially when someone else co-work with you and changes the code remotely or you frequently update your local code from a repo.

3. In large projects, there are always many submodules, each stays in a sub folder of its own. Usually there are "private headers" in the same sub folder. And there are some common headers, possibly in one or several upper/top level include folder(s). To speed up the code parsing for a large project, instead of parsing all the code sequentially, maybe it is better to parse the common headers first. And when the user opens a source file in a sub folder before parsing the whole project is completed, it could be better to pause the global parsing and parse the header files in the same sub folder first. In this way it will be faster to enable intellisense and symbol jump after you open a random source file.

Please consider this. Thank you.

[Leedehai]

Agree.
The 2nd point - "Force Re-parsing" is ESPECIALLY a much needed feature! Sometimes the error squiggles still exist after fixing the problem, and these "unhinged" squiggles even move to a segment of code text that has no problem (the error a squiggle is used to underline no longer exists, so that squiggle becomes "unhinged").
I have to relaunch VSCode to git rid of them.

[bobbrow]

@Leedehai, if you are referring to the error squiggles from the new IntelliSense engine, we trigger a full reparse of the file after 10 seconds of inactivity, if you save the file, or if you change the editor focus to a different code file. The trigger resets the next time you edit the file. Do you still see squiggles after one of these triggers happens? I sincerely hope that you don't have to relaunch VS Code to clear them.

[sean-mcmanus]

1. We have to plans to improve the status indicator, but I'm not sure if we'll add a percentage or more detailed info yet.
2. We also plan to add a reparse command for the Tag Parser (renaming the databaseFilename property is the current workaround). But as Bob said, the IntelliSense/errorSquiggles shouldn't need a reparse command. You can tell our parser to parse the common headers first via putting them first in the browse.path list.
3. When a source file is opened, we do a parse of the file and the included files (in parallel with the global search parsing); however, it's possible the wrong include file (with the same name but different path) gets chosen to parse early during the open file scenario.

[xueyunquan]

@bobbrow @sean-mcmanus The parser could have worked and should have worked as you said, but it doesn't always do so for me. I downloaded a copy of source code of the latest Linux kernel and open it in VS code. I opened a random .c source file and waited for 12 hours, and when I hover the mouse pointer over a symbol in the file it still shows "Loading...", and it still fails to jump to any symbol outside the file.

This is unseen in small projects. Hence the suggestions.

[sean-mcmanus]

@xueyunquan "Loading..." means the main thread is stuck for some reason. It isn't related to the main parsing thread or symbols not being in the database, i.e. the symbols could be in the database, but the main thread is blocked from running the "Go to Definition" code. What is your intelliSenseEngine setting? Setting it to "Tag Parser" might fix it, if the "Default" engine is causing a freeze. If you can get it to not say "Loading..." on hover and "Go to Definition" gives no results, then that would mean the symbol is not being parsed. Can you provide a link to the Linux kernel you are using?

[Leedehai]

@bobbrow Hi thanks for your reply. Yes I am referring to the new IntelliSense engine.
The good news is, the triggers work (at least in my testing)! Thanks for pointing this out.

Edit: ..occasionally it still fails.. I tried to save the file or change focus to force a reparsing but the squiggle didn't go away. After relaunching VSCode it finally disappeared.
This squiggle indicated a "too few arguments" error, by the way, because I had a function call that does not match its declaration (which is in a .h file) and its definition (which is in the same .cpp file). My solution is not changing the function call, though, but change the function declaration AND definition.

[bobbrow]

@Leedehai, I'm moving your issue to #920. Let's resume the discussion over there. Thanks.

[xueyunquan]

@sean-mcmanus : I'm using the "Default" setting for the intelliSense. An example set of code is: https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/releases/tag/roc-1.6.0

Opening this (particular version of) code in VS Code causes the parser stop working from time to time, especially after long time of browsing (opening & closing different files). Sometimes the parser would resume working if I close and restart VS Code, but after some time it might still show "Loading..." upon hovering of any symbol. Maybe the parser is too tired after hours of working? :)

[Colengms]

Hi @xueyunquan . Are you still seeing this issue? From your description, it sounds like IntelliSense parsing was failing due to encountering an issue when parsing certain code. Since your last response, there have been quite a few fixes to IntelliSense parsing. I'm hoping the issue you were seeing may have since been addressed.

[andrewmcdonald-oxb]

Hi, did these suggestions make it in? I'm in a situation where tag parsing appears to be going nowhere, with a large project loaded (~950,000 loc). I can hover over symbols and see a popup with the type info, which I understand is coming from the newer Intellisense engine. But when I right-click and choose "Go to definition" it always says "No definition found" (even for simple functions in the same file) which if I'm correct is supposed to be the job of the tag parser. There's permanently a little cylinder icon in the status bar, hovering over which always shows "tag parser initializing". Clicking it produces a "Pause Parsing" command, but I can't tell whether clicking this does anything. How do I know what's going on in this situation? I'm running VS Code 1.46.1 with cpptools 0.28.3 and CMake Tools 1.4.1, which is set as the configuration provider in C++ settings. I'm on Mac OS Catalina 10.15.5. Feel free to redirect me somewhere else if there's a better way of getting support for this

[boocs]

But when I right-click and choose "Go to definition" it always says "No definition found" (even for simple functions in the same file) which if I'm correct is supposed to be the job of the tag parser. There's permanently a little cylinder icon in the status bar, hovering over which always shows "tag parser initializing".

This info is from Windows 10, VS Build Tools 2019
Know that some of this info I've only found that last few days so it could be wrong. Haven't thoroughly tested.

The only thing I see the Tag Parser doing is Alt+O (Switch from header to source) and Go to Definition.
You can check out what your Default version of 'Intellisense' is capable of by disabling the Tag Parser and see what you're missing. It seems the same, according to your above post, since your Tag Parser hasn't finish yet.

To disable the Tag Parser you need to feed it an empty array for the includes. I do it in my settings.json. You'll need to set this for each workspace (unless you do it globally in your user settings or project wide in your project json file). Depending on where you have this setting set, it'll override other config files so be aware.

"C_Cpp.default.browse.path" : []

To see if it's working you can set this:
"C_Cpp.loggingLevel": "Debug"
Now open your Output and change it to C/C++
Open up a config file like settings.json so you don't see any Default 'Intellisense' logging. You might as well restart VSCode to.
You shouldn't see any of these lines below in your Output logging (On Windows 10 you will see a few since some are automatically included):

Processing folder (recursive): ...

Now if you test without the Tag Parser and it's correct that all that you are missing is Alt + O and Go to Definition then you really only need to include Source file paths and leave out any header directories in the "C_Cpp.default.browse.path".
I've tested this on the Unreal Engine (non full source) and it does work. The Unreal Code does a good job of separating headers and source files so it's nice.

Now if your project doesn't separate source and headers then you could just add the most used class directories at first and then add folders as you find things where Go to Definition doesn't work.

With UE4 my Tag Parser file went from 2 Gigs to 200 MB. Also typing in a source file function doesn't really trigger any Tag Parser or Default Intellisense icons so I'm guessing it's doing it in memory. In source files, Intellisense and Snippets seem pretty snappy.

It's also of note to know how the Tag Parser works. If you don't specify the Tag Parser include setting then it'll pull from your default Intellisense 'includePath' setting.
BUT... It'll also add your workspace folder automatically to your Tag Parser includes list. So for every workspace you have it'll recursively go through every directory and add every single thing.
After it does this it combines everything in one list and creates one huge Tag Parser file.

So at least for Windows 10, VSBuild Tools 2019:
Default Intellisense 'includePath' setting should be header file directories.
Tag Parser Intellisense browse path setting should be source file directories.

Also of note:
For Tag Parser includes you can disable recursive searching by adding * to the path:

"C_Cpp.default.browse.path": [
        "${workspaceFolder}/Source/*"
    ]

The Default 'Intellisense' include path automatically doesn't search recursively. You can add a ** to do so:

"includePath": [
                "${workspaceFolder}/**"
            ],

This will force to search recursively so be careful on where you use it.

One last Edit:
In your c_cpp_properties.json you can have multiple configurations. So you could have a minimal tag parser config and a more thourough config. Each config would need a separate:

"browse": {
    "databaseFilename": "..."
}

[sean-mcmanus]

Hi, did these suggestions make it in? I'm in a situation where tag parsing appears to be going nowhere, with a large project loaded (~950,000 loc). I can hover over symbols and see a popup with the type info, which I understand is coming from the newer Intellisense engine. But when I right-click and choose "Go to definition" it always says "No definition found" (even for simple functions in the same file) which if I'm correct is supposed to be the job of the tag parser. There's permanently a little cylinder icon in the status bar, hovering over which always shows "tag parser initializing". Clicking it produces a "Pause Parsing" command, but I can't tell whether clicking this does anything. How do I know what's going on in this situation? I'm running VS Code 1.46.1 with cpptools 0.28.3 and CMake Tools 1.4.1, which is set as the configuration provider in C++ settings. I'm on Mac OS Catalina 10.15.5. Feel free to redirect me somewhere else if there's a better way of getting support for this

Your database appears to be locked by a dangling cpptools process. Killing all the cpptools processes should fix that.