Vscode-cpptools: Scope is global in long function

[LeThiHyVong]

Environment:

• OS: Windows 7 SP1 32 bit
• VSCode: 1.19.2
• C/C++ extension: 0.14.6-beta1

Reproduce:

1. Open a long code file (~30k lines), wait for the database indexing
2. Move cursor to inside of a long function (~6k lines)
3. Check the scope and function list (Ctrl+Shift+O)
   -> The scope is global, the function list didn't have the current function

Reproduce without other extension: YES

[sean-mcmanus]

Thanks for reporting this failure. It appears to fail to parse symbols in the file. Can you provide more repro details? Are any symbols available in the file? Does the navigation appear near the beginning of the function (your repro mentions ~6k lines deep)? It should have nothing to do with the length, because we in general we work with 60k+ long functions and 200k+ line files, but there could be code somewhere that's messing up the parser -- but we do have a 1000 function limit you might be hitting? Do you know if this bug repros with 0.14.5? I assume it would, since we didn't change how individual files are parsed, only the method of which files are scanned for parsing. Also, the database indexing icon only shows status for the non-open files, so theoretically a large open file may still be being parsed. The method I've used previously to identify the source of failures like this is binary search (i.e. divide the code in halves or more until the failure stops and then divide in smaller increments). Macros/attributes missing from cpp.hint can cause parsing failures (https://github.com/Microsoft/vscode-cpptools/blob/master/Extension/cpp.hint ).

[LeThiHyVong]

Sorry for late response.
In that file, some simple and short symbols in that file still be displayed and navigated correctly in function list (I choose shorter function below and above example function and they are still be correct). Even inside the failed function, the go to definition etc. still works.
I just randomly choose the some long function and it failed (on other long files, too).
The common features of these failed functions are they have deep predirectives and complex call graph (13 layer exactly).
I can't reproduce in 0.14.5 since it failed to parse long code files, remember?

[sean-mcmanus]

@LeThiHyVong Does the long function signature appear in the list of available navigation targets when you click the (Global Scope) in the bottom? I assume the list doesn't have 1000 entries. It's possible we may need a repro or we'll need to add better logging for possible failures in this scenario. The bug will repro if our parser somehow thinks that a local function is defined inside the function:
int func() { int func2() {} }
or if a } is #ifdef'd away, because our symbol parser will always process the #ifdef code (this seems like the most likely cause).
`

ifdef NEVERDEFINED

}

endif

`
Basically, if a } is detected by our parser that matches the function's starting { it could count it ending the function, entering the global scope afterwards. Looks like VS 2017 has the same problem.

[LeThiHyVong]

@sean-mcmanus No, it's totally disappeared in navigation target.
Seem like it's the second case. There are a lot of similar predirective block, but it inside the line comment (// }).
Due to my company security rules, I can't give you the entire source code.
But our commenting styles are:

1. New code:

/* +++<Number> */
// new code here
/* ---<Number> */

1. Remove code:

#if 0 /* +++<Number> */
// old code will be line-commented
// printf("Remove code");
#endif /* ---<Number> */

1. Replace code

#if 0 /* +++<Number> */
// old code will be line-commented
// printf("Remove code");
#else
printf("New code");
#endif /* ---<Number> */

I'll try VS2017 and C/C++ extension 0.13.0 when I back to work at Monday.
Thanks.

[LeThiHyVong]

@sean-mcmanus
ok, After several trial, it seems your second suggestion is right.
When I use Ctrl+Shift+\ at the failed function opening bracket, it jump to the wrong closing bracket.
After removing all predirectives (the function still be 5k lines), it jump to right close bracket, but scope still global inside the function.
From that failed function, no matter what before or after removing predirectives, the scope of inside all below functions are all global and they are not listed in the function list. _(sorry for wrong information, I've updated previous comment)_
I've checked, VS2017 15.5.2 parse the file correctly (function list, navigation, scope), C/C++ extension 0.13.0 didn't (same as 0.14.6-beta1).

[sean-mcmanus]

So you're saying in VS 2017 the top right shows the function name instead of being blank, while we show "(Global Scope)"? It sounds like the functions I'm not sure how we can fix this without finding a repro.

[LeThiHyVong]

Below is the simplified source code that has the bug.
You can see the last function, ErrorFunc, has global scope inside.
sample.zip

[sean-mcmanus]

Thanks a lot for the repro -- the bug is caused by our incorrect handling of the UTF-8 characters > 1 byte. Ironically, I was already working on UTF-8 > 1 byte fixes, although they were in regards to identifier characters and not strings. We'll try to get a fix for the February update.

[sean-mcmanus]

Actually, the cause for the failure was the fact that UCHAR is undefined. As soon as I add "typedef unsigned char UCHAR" then the "UCHAR ErrFunc" parses successfully. This behavior is "by design". Do you have another repro. The good news is that the UTF-8 > 1 char handling is correct in this case. Let us know if you think there's some change we can make to handle this better. IntelliSense will generate an error for the undefined UCHAR (ideally we would have some way to expose the symbol parser failures too, but we don't).

[LeThiHyVong]

It's weird.
When I replace each 'ト' character to "tt" in all file (I suppose the ト is 2 bytes) then the parser also working correctly without a definition of UCHAR. (see sample_replaceト.c in attachment)
However, when I put UCHAR definition (like your) on the top of sample.c, the problem still persists. (see sample_uchardef.c in attachment).
Hope it helps.
sample.zip

[sean-mcmanus]

Hmm, okay, sounds like there's still a UTF-8 related problem -- I'll investigate more.

[sean-mcmanus]

What encoding is that file supposed to be? VS Code is opening it as UTF-8, but as soon as I make a change and save the file, it appears to modify the string characters somehow and the bug goes away. It's possible our internal document format is corrupting the data...

[sean-mcmanus]

Wow -- it looks like once the file is modified/saved, the encoding changes are not undo-able using VS Code. The character changes are being done by VS Code (not our extension), which suggests that the file originally was not meant for UTF-8 encoding? The original characters cause our tag parser to fail, causing the ErrFunc to not be found. It doesn't repro in VS 2017 because they do some extra code page conversion. I don't think this is fixable until we add support for non-UTF-8 encodings (the workaround is to make sure the file is encoded using UTF-8).

[LeThiHyVong]

Sorry for not mention, the file encoding is Shift jis.

[LeThiHyVong]

@sean-mcmanus Any update?
The bug still persists on latest extension (0.17.0).
The file encoding is SHIFT JIS.
When I reopen the file, convert it to UTF-8 or replace all Japanese characters, the file is parsed correctly.
Thanks.

[sean-mcmanus]

We haven't gotten around to fixing our UTF-8 bugs. I'm not sure of an ETA yet.

[bobbrow]

We are tracking support for non-utf8 encodings with #414

[sean-mcmanus]

The Shift JS issues should be fixed with https://github.com/microsoft/vscode-cpptools/releases/tag/0.30.0-insiders4 . Let us know if you see any remaining bugs with that encoding.