Vscode-cpptools: clang-format corrupting files?

[pbnjay]

This is the strangest bug I've seen, very intermittent but getting more common on my system, even after restarting VS Code and/or rebooting. I have no idea how to make a reproducible test case so hopefully someone has an idea what is going on.

When I save the C file I'm working on, the file is very strangely corrupted (probably due to the clang-format in some way). I have a short video where all I do is hit save then undo a couple times, and it shows that even the corruption is not consistent from save to save.

I just updated everything this morning but it's still happening...
VS Code Insiders build 1.17.0
C/C++ extensions version 0.13.0

Only 2 other extensions installed: Sublime Text Keymap 2.9.1 and Go 0.6.65

Entirety of my
settings.json.txt

On macOS 10.12.6 (16G29)

[sean-mcmanus]

We have seen file corruption on formatting in the past when UTF-8 characters > 1 byte long have been used, but we thought we fixed all those cases. Do the corrupted files have non-ASCII UTF-8 characters? Have you ever reproed the bug with clang-format itself via the command line? You could also try upgrading to a newer version of clang-format. I would recommend not to use "C_Cpp.clang_format_formatOnSave": true and instead use the editor.formatOnSave setting (our setting has some known bugs).

[pbnjay]

I disabled the clang formatOnSave and tried just doing it manually (shift-option-f) and that also showed corruption. On the command line (up-to-date via brew install clang-format), I did not see any corruption.

It's not doing it now, but I'll switch to editor.formatOnSave and see if it happens again.

Shouldn't be any UTF-8 issues, max char appears to be 126 via python -c 'print max([ord(c) for c in open("test.c").read()])'

[sean-mcmanus]

@pbnjay FYI, the corruption previously has been in our internal "document" representation for the file that we pass into clang-format and not the actual formatting process (which we get from clang-format), so if our formatOnSave setting lead to corrupting the document, then you would need to do a Reload Window before manually formatting would give correct results.

[pbnjay]

Well it came up again, so switching the formatOnSave option didn't help any.

"C_Cpp.clang_format_formatOnSave": false,
"C_Cpp.clang_format_fallbackStyle": "LLVM",
"C_Cpp.clang_format_style": "LLVM",
"C_Cpp.clang_format_path": "/usr/local/bin/clang-format",
"editor.formatOnSave": true,

FWIW, it looks like it might be random bits from my copy-paste buffers that are being flung around. Should I switch back to a non-insiders build or is this going to still be an issue there?

[sean-mcmanus]

If it's a VS Code bug, non-Insiders might help, but our bits are the same either way. Can you try setting C_Cpp.formatting to "Disabled" and using another extension for formatting or invoking formatting via the command line? We need to figure out a repro so we can fix this. This is the only report we've received of document corruption recently. Can you think of anything unusual in your scenario, such as UTF-8 characters > 1 byte being in your path? Do you know if this is a regression from a previous build? I don't recall any changes recently to this area of code.

[sean-mcmanus]

Is this a problem with no break space 0xA0 in your document: https://github.com/Microsoft/vscode-cpptools/issues/1082 ?

[pbnjay]

Nope. Maximum character is 126 (0x7E / "tilde") minimum character value is a newline \n.

[pbnjay]

Also, I only discovered/enabled clang-format on the 22nd, so I can't say for sure but I had no issues until the 26th or so. And I was probably using it 10h+ per day in that stretch. So looks highly likely to be a regression somewhere at least.

[bobbrow]

So far there are reports of this on Mac and Linux. If anyone is seeing this on Windows, please let us know. We're actively trying to reproduce this but haven't had any success yet,

[SE2Dev]

I'm on Debian Stretch, and had the formatter break while attempting to debug the following program with GCC and a basic makefile:

#include <iostream>

template<typename T>
struct foo
{
    T v;

    constexpr foo(long double val) : v((T)val) {}

    friend constexpr foo operator"" _deg(long double deg);
};

// used as conversion
constexpr foo<float>
operator"" _deg(long double deg)
{
    return foo<float>(deg);
}

int main()
{
    double x = (90.0_deg).v;
    std::cout << std::fixed << x << '\n';
}

[aytekinar]

Same here on Arch Linux.

clang-format -version
clang-format version 5.0.0 (tags/RELEASE_500/final)

My main.cpp file is as follows:

#include <iostream>

template <class T>
class MyClass {
 private:
  T val_;
public:
  MyClass() = default;
  MyClass(T val) : val_{val} { }
  void setval(T val) { val_ = val; }
  T getval() const { return val_; }
};

int main(int argc, char *argv[]) {
  MyClass<double> myobj{5};
  myobj.setval(6);
  std::cout << "val: " << myobj.getval() << '\n';
  return 0;
}

VSCode's user settings are simply:

{
    "workbench.colorTheme": "Solarized Dark",
    "julia.executablePath": "/opt/julia-0.6/bin/julia",
    "editor.tabSize": 2,
    "editor.fontFamily": "'Source Code Pro', monospace",
    "editor.fontSize": 14,
    "C_Cpp.clang_format_path": "/usr/bin/clang-format",
    "C_Cpp.clang_format_fallbackStyle": "LLVM",
    "C_Cpp.intelliSenseEngine": "Default"
}

and the output to /usr/bin/clang-format -style "LLVM" main.cpp is:

#include <iostream>

template <class T> class MyClass {
private:
  T val_;

public:
  MyClass() = default;
  MyClass(T val) : val_{val} {}
  void setval(T val) { val_ = val; }
  T getval() const { return val_; }
};

int main(int argc, char *argv[]) {
  MyClass<double> myobj{5};
  myobj.setval(6);
  std::cout << "val: " << myobj.getval() << '\n';
  return 0;
}

I am running VSCode on commit Microsoft/vscode@27492b6b with cpptools at 0.13.0.

[sean-mcmanus]

Can someone who is able to repro this bug use 0.13.1, and enable logging with level >= 5 (https://github.com/Microsoft/vscode-cpptools/blob/master/Documentation/LanguageServer/Enabling%20logging.md ) and let us know if the corruption first appears in the "Formatting input: ", "Formatting raw output", or "Formatting diffed output" sections?

[aytekinar]

@sean-mcmanus,

Please find below the corresponding parts of the log:

Formatting input: #include <iostream>

template <class T>
class MyClass {
private:
  T val_;
public:
  MyClass() = default;
  MyClass(T val):val_{val}{}
  void setval(T val) { val_ = val; }
  T getval() const { return val_; }
};

int main(int argc, char *argv[]) {
  MyClass<double> myobj{5};
  myobj.setval(6)   ;
  std::cout << "val: " << myobj.getval()<< '\n';
  return 0   ;
}
Formatting document: file:///home/aytekin/Desktop/cpp-tryout/main.cpp
    Formatting raw output: #include <iostream>

template <class T> class MyClass {
private:
  T val_;

public:
  MyClass() = default;
  MyClass(T val) : val_{val} {}
  void setval(T val) { val_ = val; }
  T getval() const { return val_; }
};

int main(int argc, char *argv[]) {
  MyClass<double> myobj{5};
  myobj.setval(6);
  std::cout << "val: " << myobj.getval() << '\n';
  return 0;
}
    Formatting diffed output:  class MyClass {
private:
  T val_;

public:
  MyClass() = default;
  MyClass(T val) : val_{val} {}
  void setval(T val) { val_ = val; }
  T getval() const { return val_; }
};

int main(int argc, char *argv[]) {
  MyClass<double> myobj{5};
  myobj.setval(6);
  std::cout << "val: " << myobj.getval() << '\n';
  return 0;

The bug seems to have gone. However, since my last post yesterday, and just prior to your message 10 hours ago, I have received two updates: (1) from vscode, and (2) from cpptools.

Now I am running:

code --version
1.17.0
be377c0faf7574a59f84940f593a6849f12e4de7

as well as cpptools 0.13.1. Basically, I cannot confirm if the bug would have existed in either vscode or the extension in the previous version pair. Now, I do not see any bug.

[SE2Dev]

I can confirm that the issue still occurs with the latest version of VSCode (1.17.0) and the extension (0.13.1).

code --version
1.17.0
be377c0faf7574a59f84940f593a6849f12e4de7

My guess would be that it's related to how the plugin handles parsing the code - because I noticed this time that the corrupt formatting data largely consists of a chunk of code that I wrote seconds beforehand. Perhaps there's a race condition between parsing & formatting?

[sean-mcmanus]

@SE2Dev Could you enable the logging we added and tell us at what point the data is corrupted (as input, raw output, or diffed output)? Also, when you undo the formatting to get back the original text, do other language service features still work, like hover or error squiggles? A race condition would only happen if multiple threads modify the document, but only the main thread applies edits to the document and not the parsing threads. It sounds like some memory corruption. Our internal document can contain old text in "unused" sections of memory, which sounds like what you are seeing.

[SE2Dev]

@sean-mcmanus I enabled logging this morning just after I confirmed that the issue still occurs in the current version, but it hasn't manifested again since I enabled logging. I'll be sure to update with logs when it happens again.

[pbnjay]

FYI I haven't seen any issues with 0.12.4.

I've switched back to 0.13.1 and enabled logging, I'll report back if I see anything.

I recorded a short video of the corruption with 0.13.0 (as mentioned in the original report) and uploaded it here: https://youtu.be/c-4LSWeE1QI

[sean-mcmanus]

@pbnjay We weren't able to identify any regression with the 0.13.0 changes that could cause this (we didn't make any formatting or document changes). It's possible that 0.12.4 has the root bug too but memory has just shuffled around such that it doesn't repro as much.

[SE2Dev]

So unbeknownst to me - I apparently have been reproducing the issue, or a related issue, repetitively over the past few days. There are some distinctions between the two however; The first one results in small or large scale corruption of the file data upon formatting, the second simply replaces all current text with a copy of itself (as though formatting didn't yield any results).

I managed to catch this behavior a few moments ago and recorded a video of it. Additionally, I had logging enabled, so I'll include the logs here as well.
vscode.cpp.log.3552.txt (90 bytes)
vscode.cpp.log.31889.txt (1.6 kb)
vscode.cpp.log.31703.txt (9.9 kb)
vscode.cpp.log.31317.txt (1.9 kb)
vscode.cpp.log.31223.txt (282.6 kb)

[sean-mcmanus]

@SE2Dev The last log is very helpful. The root bug is not with formatting (as I suspected), but with the internal document representation being corrupted (which will cause most of our features to behave incorrectly). Now we just have to figure out what operation is causing the corruption (formatting has been excluded because the input to formatting is corrupt already). The only event that intentionally changes the document is the didChange event, so either there's a bug with how edits are applied to the document or this section of memory is randomly being corrupted by some other operation.

[aytekinar]

Maybe this is not the place, as the thread is about the corruption of the formatting, but as I have posted under #525, the "Format document" option does not work after some time of use.

[SE2Dev]

I haven't been able to reproduce the issue very often (if at all) as of the most recent vscode-cpptools update (0.14.0).

[aytekinar]

@SE2Dev, so far so good for my case, too. It seems that either the update to VSCode or to cpptools (or both) has solved the problem.

[WillEwing]

I'm seeing this happen on VSCode 1.17.2 / Cpptools 0.14.0. It seems to start when I try to format a source file I have open in multiple tabs.

[WillEwing]

A further detail I'd missed: it's much more likely to happen if the file has been edited in multiple tabs with different capitalization of the file name on Windows. This tends to happen because #272 makes the debugger open an all-lower-case version of files with a mixed-case names even if they're already open.

[sean-mcmanus]

@WillEwing Thanks a lot for that info. I have finally been able to repro the bug via doing edits to the 2 files with different cases. If people are hitting this with other scenarios, then hopefully the root cause will be the same.

[bojunehsu]

If it helps, I also encountered this issue before on Windows. https://github.com/Microsoft/vscode/issues/36418. I don't recall having multiple tabs with different casing though. I will enable logging and share if I encounter the issue again.

[sean-mcmanus]

Has the document corruption after formatting been fixed? An internal user is reporting a problem still with 0.14.6, but we fixed some crashes in our preview .vsix https://github.com/Microsoft/vscode-cpptools/releases/tag/v0.15.0-insiders that could be causing a problem.

[aytekinar]

I haven't had any issues since the last update under this thread.

[winterscar]

I'm experiencing this issue on windows 10 with the latest build at the time of writing. Anything I can help with?

[sean-mcmanus]

@winterscar It's most likely a corrupt of our internal document object (i.e. the root cause occurs before the formatting and the formatting makes it visible). You can set the loggingLevel to the hidden value of "6" to view the formatting input/output for debugging. Are you able to determine which actions are causing the corruption? We have internal debug flags that can be turned on to catch these document corruptions, but nothing that is user-accessible currently.

[samdenty]

I've been having this issue persistently until a manual restart of VS code, along with broken intellisense. Formatting works fine at first, then starts garbling up the code. I can format unopened files fine before it breaks.

Tested in VS 1.29.1+

Original code (before format document was called)


[mmichon]

Same here. VScode 1.30.1 on Mac Mojave, clang-format 8.0.0.

[sean-mcmanus]

@mmichon Can you try with our latest 0.21.0-insiders3? It has some changes that may have fixed this. Otherwise, we'd love to fix this this, but we need a repro or more debug logs to help us track down the cause.

Setting C_Cpp.updateChannel to "Insiders" is another way to get the update.

[mmichon]

@sean-mcmanus I'm running 0.21.0-insiders3 right now and turning logs up to DEBUG and it just recurred. I have a before/after corruption diff that I can attach if it's useful. I don't know where extension logs live on Mac but I did see a vscode-cpptools reference in an error in the Dev Tools Console logs:

AssertionError [ERR_ASSERTION]: false == true at Console.assert (console.js:191:23) at DidCloseTextDocumentFeature.didClose [as _middleware] (/Users/mmichon/.vscode/extensions/ms-vscode.cpptools-0.21.0-insiders3/out/src/LanguageServer/protocolFilter.js:44:25) at DidCloseTextDocumentFeature.callback (/Users/mmichon/.vscode/extensions/ms-vscode.cpptools-0.21.0-insiders3/node_modules/vscode-languageclient/lib/client.js:253:22) at e.fire (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:133:238) at /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:537:143 at e.fire (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:133:218) at e.$acceptDocumentsAndEditorsDelta (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:541:607) at t._doInvokeHandler (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:658:837) at t._invokeHandler (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:658:509) at t._receiveRequest (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:657:126) at t._receiveOneMessage (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:656:59) at /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:653:895 at /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:130:793 at e.fire (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:133:218) at a (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:186:147) at Socket.n._socketDataListener (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/node/extensionHostProcess.js:186:368) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10) at Pipe.onread (net.js:594:20)

[sean-mcmanus]

@mmichon You can enable Debug logging. With debug logging we show the inputs/outputs of the formatting -- my guess is that the inputs are incorrect due to some bug with the editing of our internal document object.

Are you using multiple workspace folders? Does the bug repro when using only 1 workspace folder?

I have never seen that assertion error before, but it seems unrelated.

If you can find some sort of simple repro for us that would be great, i.e. what sort of pre-requisite steps are required?

[mmichon]

@sean-mcmanus logging had been set to Debug before the last repro. Where can I find the logs?

Not sure about repro steps yet -- seems random right now. I do have multi-byte Unicode (emoji in strings) scattered throughout the code though. I think it started when I added that.

[sean-mcmanus]

@mmichon The logging is shown in the C/C++ Output pane in VS Code.

Your files are UTF-8 encoded, right? If you could provide a small test file with the particular emoji characters that might help. We have UTF-8 tests with 3-4 byte characters, but maybe we're missing some special case. I've tried formatting with бв包括, but I haven't observed the bug yet.

Bob said previously in https://github.com/Microsoft/vscode-cpptools/issues/2731 that he thought it could be Terminal or Output window related.

[giellamoswhard]

Sorry I've not read all the issue but in my case clang format corruption was due to the presence of symbolic link inthe folder path.

[mmichon]

@sean-mcmanus yes, they're UTF-8. Here's a version of it. Note the emoji in line 126.

Re: @giellamoswhard's comment -- this file lives in a symlinked path as well.

sample.cpp.zip

[sean-mcmanus]

@mmichon I tried editting/formatting that sample.cpp on Mac, but I'm still not able to repro any formatting corruption (I was using 0.21.0-insiders3) -- are there some additional steps or special settings I need? On Linux, I'm able to have a symlink on the path, but I still don't repro any formatting corruption, and on Mac the OS appears to resolve the symlink when I do an Open folder so I don't know if there's some other way to make the symlink appear in the path?

Is anyone able to check their formatting input/out logs? That would at least tell us if the formatting corruption is due to the input being corrupt, or the formatting causes it, or some post-processing on the formatting is causing it.

[darkfader]

I got corruption too but after clearing my debug output and saving again, it gave no error.

[Error - 1:41:21 AM] Request textDocument/documentSymbol failed.
Error: Connection got disposed.
    at Object.dispose (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/main.js:825:25)
    at Object.dispose (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-languageclient/lib/client.js:57:35)
    at LanguageClient.handleConnectionClosed (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-languageclient/lib/client.js:1864:38)
    at LanguageClient.handleConnectionClosed (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-languageclient/lib/main.js:106:15)
    at closeHandler (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-languageclient/lib/client.js:1852:18)
    at CallbackList.invoke (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/events.js:71:39)
    at Emitter.fire (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/events.js:135:36)
    at closeHandler (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/main.js:221:26)
    at CallbackList.invoke (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/events.js:71:39)
    at Emitter.fire (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/events.js:135:36)
    at StreamMessageReader.AbstractMessageReader.fireClose (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/messageReader.js:135:27)
    at Socket.<anonymous> (/Users/me/.vscode/extensions/ms-vscode.cpptools-0.20.1/node_modules/vscode-jsonrpc/lib/messageReader.js:188:62)
    at emitOne (events.js:121:20)
    at Socket.emit (events.js:211:7)
    at Pipe.Socket._destroy.cb._handle.close [as _onclose] (net.js:554:12)

clang-format version 8.0.0 (tags/google/stable/2018-10-04)
macOS 18.2.0

[sean-mcmanus]

@darkfader That "Connection got disposed" means our Microsoft.VSCode.CPP.Extension process crashed. Our Insider release at https://github.com/Microsoft/vscode-cpptools/releases/ has many crashes fixed, so it's recommended that you use that if you're experiencing crashes (or via setting the C_Cpp.updateChannel to "Insiders"). The logging that would help us with the corruption issue is in the C/C++ output window.

Oh -- we haven't done any testing with clang-format 7 or 8 yet -- it's possible a newer version could somehow cause a break with us. Do these corruption issues happen with the default clang-format 6.0 that we ship with?

[mmichon]

@sean-mcmanus I haven't been able to repro and manage to catch logs yet. Re: clang [-format] version, my C_Cpp.clang_format_path was set to one that was clang-format version 8.0.0 (tags/google/stable/2018-10-04), so I'll try downgrading.

[sean-mcmanus]

@mmichon We'll probably upgrade our default clang-format to 8.0 after it ships -- we haven't bothered with 7.0 yet (we have to do some additional signing and testing), but if we're somehow incompatible with clang_format 8, we can probably fix that earlier. If there are UTF-8 characters > 1 byte in the path, that might be an issue, as reported at https://github.com/Microsoft/vscode-cpptools/issues/3007, but I'm not able to repro that yet.

[mmichon]

Just recurred on stock 0.21.0-insiders2 version. I'm including my settings.json, because I don't see any output except for (a probably unrelated) Error: no handler found in vscode's Output pane:

{
    "editor.cursorStyle": "block",
    "window.zoomLevel": 1,
    "telemetry.enableTelemetry": false,
    "editor.minimap.enabled": false,
    "editor.fontFamily": "Monaco, Menlo, 'Courier New', monospace",
    "workbench.colorTheme": "Default Light+",
    "editor.scrollBeyondLastLine": false,
    "files.trimFinalNewlines": true,
    "files.trimTrailingWhitespace": true,
    "terminal.integrated.cursorBlinking": true,
    "terminal.integrated.rendererType": "dom",
    "python.pythonPath": "/Users/mmichon/.platformio/penv/bin/python",
    "C_Cpp.updateChannel": "Insiders",
    "C_Cpp.loggingLevel": "Debug",
    "files.autoSave": "afterDelay",
    "files.autoSaveDelay": 10000,
    "editor.formatOnPaste": true
}

It's funny -- it's attaching previously pasting data (with format-on-paste enabled above):

void close_garage() {
    if (is_garage_open()) {
        hit_garage_button();
    }
}

Pasting in String timestamp = get_timestamp(); after hit_garage_button(); results in:

void close_garage() {
    if (is_garage_open()) {
        hit_garage_button();

String timestamp = get_timestamp();
String timestamp
    Error : no handler
}

Notice how I had Error: no handler in my clipboard previously. Maybe this is a vscode clipboard bug.

I can readily repro this right now, so if I get guidance on the debug logging issue soon, I can hopefully provide logs.

[sean-mcmanus]

I'm still not reproing the bug with the editor.formatOnPaste setting. What do your logs say?

I get the following:

textDocument/rangeFormatting
Formatting input: void close_garage() {
    if (is_garage_open()) {
        hit_garage_button();String timestamp = get_timestamp();
    }
}
Formatting document: file:///....cpp
Formatting raw output: void close_garage() {
    if (is_garage_open()) {
        hit_garage_button();
        String timestamp = get_timestamp();
    }
}
Formatting diffed output: 
        String timestamp = get_timestamp();

I'm using the same settings as you as far as I can tell.

Are you sure you're using insiders2 instead of insiders3? If you have updateChannel set to "Insiders" you should be getting insider3 and not insiders2, unless we have some bug with out updater. insiders3 should be higher quality and pretty close to what we plan to ship for 0.21.0, with 3 or so additional fixes.

What do you mean by "Error: no handler" being in your Output pane? Which tab? The C/C++ one? "No handler" is not a message we output.

[sean-mcmanus]

@mmichon Do you have another formatting extension installed? Such as the clang-format one? The other extension could be causing the problem and/or interfering with ours. Do are you using multiple root folders (the multiroot folder scenario can have issues when we don't get the correct messages to right folder).

[mmichon]

@sean-mcmanus I turned off all extensions except for ms-vscode.cpptools (you're right, I'm running .021.0-insiders3), akamud.vscode-theme-onedark, and platformio.platformio-ide.

Just reproduced it again. It's actively corrupting everytime I have it format. Again, the bottom pane 'OUTPUT' pane doesn't show anything except 'No handler'. OTOH, I have no idea if this is where extension logs go. Point me to a doc on where VSCode macOS extension logs go and I'll include them here.

[sean-mcmanus]

I don't see a "No handler" error message used our code, VS Code's, or platform IO's, so I don't know where that is coming from.

Info on getting our logging is at https://github.com/Microsoft/vscode-cpptools/blob/master/Documentation/LanguageServer/Enabling%20logging.md .

[mmichon]

@sean-mcmanus alright, finally got some debug output, but it doesn't look useful. This is what I see after undo/redoing after a corruption event:

tag parsing file: /Users/mmichon/Library/Mobile Documents/com~apple~CloudDocs/src/esp-house/esp-doorbell/src/main.cpp
cpptools/activeDocumentChange
cpptools/textEditorSelectionChange
cpptools/textEditorSelectionChange
textDocument/documentHighlight
textDocument/codeAction
Database safe to open
cpptools/textEditorSelectionChange
textDocument/documentHighlight
textDocument/codeAction
textDocument/didChange
cpptools/textEditorSelectionChange
textDocument/codeAction
cpptools/textEditorSelectionChange

It's weird, because the file I'm editing is in the esp-doorbell project but this is the debug output for the esp-entrance project.

If you look at line 24 before and after corruption, it looks like it's just overwriting characters in a buffer:

[mmichon]

Ok, new corruption on the same file, little more verbose:

cpptools/activeDocumentChange
cpptools/textEditorSelectionChange
cpptools/textEditorSelectionChange
textDocument/codeAction
cpptools/textEditorSelectionChange
textDocument/formatting
Formatting input: #include "config_mmichon.h"

include

include

ifdef ESP8266 // if ESP8266

include

include

endif

ifdef ESP32

include

include

define LED_BUILTIN 2

endif

include "../../common/lib/common.cpp"

define PIN_DOORBELL 14 // D5, GPIO14

define PIN_CHIME 4 // D2, GPIO4 // XXX unused

define PIN_DOOR_SENSOR 5 // D1

//#define BACKEND_HOST "192.168.0.101" // local

define S3_URL "https://s3-us-west-2.amazonaws.com"

define URL_NA "https://image.freepik.com/free-icon/not-available-abbreviation-inside-a-circle_318-33662.jpg" // N/A placeholder image

define DELAY_AUTO_PRESS 4000L // How long to wait between doorbell press and opening the gate

define DELAY_IGNORE_HW_LIGHT 5000L // How long to ignore hw light after b...

Formatting document: file:///Users/mmichon/src/esp-house/esp-doorbell/src/main.cpp
Formatting raw output: #include "config_mmichon.h"

include

include

ifdef ESP8266 // if ESP8266

include

include

endif

ifdef ESP32

include

include

define LED_BUILTIN 2

endif

include "../../common/lib/common.cpp"

define PIN_DOORBELL 14 // D5, GPIO14

define PIN_CHIME 4 // D2, GPIO4 // XXX unused

define PIN_DOOR_SENSOR 5 // D1

//#define BACKEND_HOST "192.168.0.101" // local

define S3_URL "https://s3-us-west-2.amazonaws.com"

define URL_NA "https://image.freepik.com/free-icon/not-available-abbreviation-inside-a-circle_318-33662.jpg" // N/A placeholder image

define DELAY_AUTO_PRESS 4000L // How long to wait between doorbell press and opening the gate

define DELAY_IGNORE_HW_LIGHT 5000L // How long to ignore hw light af...

Formatting diffed output: blynk_timer.setTimeout(DELAY_AUTO_PRESS, {
log_line("Auto-opening gate");

            bridge_entrance.virtualWrite(V_ENTRANCE_BUTTON, 1);
        });
    }

    //bBlynk.notify(notification);
}

/eSydd state

ssatte))

//ethuto/Opte teeseng is ot

/LesblIGHkSieeA.HING, rttet_egnureuElash_st/st/io, u//
N__st ,//io, ufo epAnno{e Auto ingn n);e);
b r setiieTToetflaetu((_EAAU_BUTTON b r,setiieTToet laetu((_EAAU_BUTTONReser,button stat
/ es nk_ rmgr s);T m ti _LRY ng g t_HOLDe re et_recently_pressed_button1t //;Reset visual state in app

≈a.u
// Open t eagaterif_theCAutoTOpen,setting is1on
ifd(auto_open)n{
≈(V_TCMtuDOO BELLadUNG, lgatatimuTtamp)≈;

    s irsoreteft shtcount(); // Rlavtetheyura hrcounterEafterewe're)do e

}

/*
oBd iCACHturAMVATT__handlO_doorRsLn_or, {
door_sensor_pressed_recently = true;
if (door_sensor_already_pressed == false) { Blynk.virtualWrite(V_TIME_DOORBELL_RUNG, ge...
terminating child process: 21269
textDocument/didChange
cpptools/textEditorSelectionChange
Checking for syntax errors: file:///Users/mmichon/src/esp-house/esp-doorbell/src/main.cpp
queue_update_intellisense for files in tu of: /Users/mmichon/Library/Mobile Documents/com~apple~CloudDocs/src/esp-house/esp-doorbell/src/main.cpp
tag parsing file: /Users/mmichon/Library/Mobile Documents/com~apple~CloudDocs/src/esp-house/esp-doorbell/src/main.cpp
sending 159 changes to server
errorSquiggles count: 17
errorSquiggles count: 0

[sean-mcmanus]

@mmichon It's a bug with our multi-root support. I've reproed the bug via editing a file that is a child of 2 folders, which is not a scenario we "support". Can you work around the issue by having all the workspace folders not be a child of another folder? We should be able to fix the format corruption, but there are other bugs in that scenario, such as https://github.com/Microsoft/vscode-cpptools/issues/2968 .

[mmichon]

@sean-mcmanus many thanks. Will do.

[nebbles]

@mmichon It's a bug with our multi-root support. I've reproed the bug via editing a file that is a child of 2 folders, which is not a scenario we "support". Can you work around the issue by having all the workspace folders not be a child of another folder? We should be able to fix the format corruption, but there are other bugs in that scenario, such as #2968 .

Eventually got here after I set up a new project in the PlatformIO. The default there is to create a new folder at the target location and add that folder to the workspace. For projects where a directory (e.g. git) is already established, it immediately induces this issue of a file being a child of two folders in the workspace. This is probably something that PlatformIO should address as a default behaviour but just wanted to mention it here.

Thank you for identifying/suggesting this 🙏

[maciejmatczak]

Hi there, I am having similar issues.

I have symlinked folders and I believe I am getting files opened in a way of file being child of two folders. While opening file via Explorer it gets symlinked path. But try to use "Go to definition" on some function: it opens next tab with an absolute file path, even it's the same file.

I am not sure who exactly handles with "Go to definition" - hopefully you will find it helpful!

[sean-mcmanus]

@maciejmatczak I don't know what you mean by "similar issues" -- you're seeing corrupted files? The symlink issue appear to be tracked by https://github.com/Microsoft/vscode-cpptools/issues/3288 (Go to declaration always shows 2 files, relative and absolute paths), so you could upvote that.

[maciejmatczak]

Hi @sean-mcmanus, excuse me if I wasn't clear! Yes, as this issue is about clang formatting code corrupting files by saying "similar issues" I meant that my files gets corrupted when I am, well, formatting the documents.

Example as easy as one folder with .cpp and .h is enough. Just make a symbolic link next to original folder and work on those files. Hit "Go to definition" (you land in file opened with absolute path, though they are the same; but this is another issue tracked in #3288 as you said). Make some edits, save, try to format the document multiple times in a row. Take this example if you want:

someLib.zip

And this is how it looks like:
https://photos.app.goo.gl/cbAwdjDVdZMHjV5z8
At first I save and format file to check if nothing bad happens. After some edits everything gets broken (sorry for this folding action in the middle, it's irrelevant for this issue).

[sean-mcmanus]

@maciejmatczak Thanks for the repro info. I was able to repro the bug on Linux. Looks like internally we have a single document buffer for edits/formatting for both documents since they're the same file, but VS Code has 2 different actual editors for the 2 paths, causing the wrong results to appear in one of the VS docs that wasn't edited. The root issue could be considered a VS Code bug, because they shouldn't be opening 2 editors for the same file.

[Victsz]

@sean-mcmanus , in my case, even only one instance of that file, this issue will still have chance to raise.
And use save without format will be just fine

[Victsz]

Is there any workaround or suggestion practice for this issue? It`s really annoying.

[sean-mcmanus]

@Victsz You could either not use multi-root workspaces or set C_Cpp.formatting to "Disabled" and use another extension for formatting.

[giellamoswhard]

@Victsz
My workaround is to stop using symbolic links in my directory structure.
@sean-mcmanus
Are we supposed to open the bug upstream to vscode ?

[sean-mcmanus]

@giellamoswhard If I remember correctly the symbolic link repro and multi-root repro are 2 different root causes that lead to same document corruption after formatting due to the didOpen not getting received for one of the documents.

I don't think VS Code is buggy, although VS Code already has bugs related to duplicate files getting opened with different casing, causing issues if our extension doesn't handle it correctly -- our next Insider release will give a warning on Windows where the same file with different casings is opened, but I don't think we added the warning to the symbolic link case yet.

[alanbirtles]

Seeing this issue without multi-root workspaces with cpptools 0.24.1, clang-format command line works correctly.

ttextDocument/formatting: 5569
Formatting input: #include "stdafx.h"
#include "TransferProcessor.h"

namespace mynamespac {

using namespace std::literals;

namespace {

struct Unlocker
{
    Unlocker(std::unique_lock<std::mutex>& lock)
      : lock(lock)
    {
        lock.unlock();
    }

    ~Unlocker()
    {
        lock.lock();
    }

    std::unique_lock<std::mutex>& lock;
};

struct CurrentTransfer
{
    CurrentTransfer(boost::optional<int64_t>& currentTransfer, Transfer& transfer)
      : currentTransfer(currentTransfer)
    {
        currentTransfer = transfer.id;
    }

    ~CurrentTransfer()
    {
        currentTransfer.reset();
    }

    boost::optional<int64_t>& currentTransfer;
};

}  // namespace

TransferProcessor::TransferProcessor(
  std::shared_ptr<IDatabase> db,
  database::TransferQuery transferQuery,
  const std::string& loggerTag)
  : db(db)
  , cancelTask(false)
  , logger(log4cplus::Logger::getInstance(loggerTag))
  , terminate(false)
  , transferQuery(transferQuery)
{}

void TransferProcessor::transferUpdate...
Formatting document: file:///....TransferProcessor.cpp
Formatting raw output: #include "stdafx.h"
#include "TransferProcessor.h"

namespace mynamespac {

using namespace std::literals;

namespace {

struct Unlocker
{
    Unlocker(std::unique_lock<std::mutex>& lock)
      : lock(lock)
    {
        lock.unlock();
    }

    ~Unlocker()
    {
        lock.lock();
    }

    std::unique_lock<std::mutex>& lock;
};

struct CurrentTransfer
{
    CurrentTransfer(boost::optional<int64_t>& currentTransfer, Transfer& transfer)
      : currentTransfer(currentTransfer)
    {
        currentTransfer = transfer.id;
    }

    ~CurrentTransfer()
    {
        currentTransfer.reset();
    }

    boost::optional<int64_t>& currentTransfer;
};

}  // namespace

TransferProcessor::TransferProcessor(
  std::shared_ptr<IDatabase> db,
  database::TransferQuery transferQuery,
  const std::string& loggerTag)
  : db(db)
  , cancelTask(false)
  , logger(log4cplus::Logger::getInstance(loggerTag))
  , terminate(false)
  , transferQuery(transferQuery)
{}

void TransferProcessor::transferU...
Formatting diffed output: 
    lock(mutex, std::defer_lock);
    if (std::this_thread::get_id() != thread.get_id()) {
        lock.lock();
    }
    if (newState == TransferState::Cancelling && currentTransfer && id == *currentTransfer) {
        cancelTask = true;
        cs_ a cs_ ctndition.notifx, sll();
        do
            oti
    }
tfyf ooUG(logger, "y_asfer 
}
void TransferProcessor::transferAmded(const Transfer& transfer)
{
    LOG4CPLUS_DEBUG(logger, "Transfer added: " << transfer.id);
    std::unique_lock < lock(mutex, std::defer_lockid);
    if (std::this_thread::get_id() != thread.get_id()) {
        lock.lock();
    }
    condition.notify_alnnd _lll();
    fy a ond
}
void TransferProcessor::transferRemoved(int64_t id)
{
    LOG4CPLUS_DEBUG(logger, "Transfer removed: " << id);
    of std::unique_lock lock(mutex, std::defer_lock);
    if (std::this_thread::get_id() != thread.get_id()) {
        cl ti lock.lock();
        anc cancalTa k{ t u{ oniconl tio t fy_all();
    }
    i
}
d = cuifn(&ur...
terminating child process: 17057
textDocument/didChange
textDocument/didChange
textDocument/didSave
  tag parsing file: ....TransferProcessor.cpp
idle loop: reparsing the active document
Checking for syntax errors: file://....TransferProcessor.cpp
queue_update_intellisense for files in tu of: ....TransferProcessor.cpp
textDocument/documentSymbol: 5571
sending 398 changes to server
errorSquiggles count: 53
Update IntelliSense time (sec): 2.064
textDocument/codeAction: 5572

[sean-mcmanus]

Our latest Insiders release has a new multi-root implementation (which fixes this bug): https://github.com/microsoft/vscode-cpptools/releases/tag/0.27.0-insiders. Please let us know if you find any multi-root-related bugs or regressions so we can prioritize fixing those for 0.27.0.

[olzhas]

Our latest Insiders release has a new multi-root implementation (which fixes this bug): https://github.com/microsoft/vscode-cpptools/releases/tag/0.27.0-insiders. Please let us know if you find any multi-root-related bugs or regressions so we can prioritize fixing those for 0.27.0.

I am affected by the bug. My working directory is a symlink, and once in a while when I reformat the code my file becomes corrupted. I am using /0.27.0-insiders2

[sean-mcmanus]

@olzhas We only fixed the multi-root cause of the corrupt files. The symlink issue is probably caused by a different issue, so I've filed https://github.com/microsoft/vscode-cpptools/issues/5061 to track that.

[olzhas]

I press "Alt-o" to switch between header and source files, and sometimes this leads to two tabs with the same source file, then I press ctrl+shift+i and file get corrupted.

[sean-mcmanus]

@olzhas Can you provide more repro details? What OS are you on? What is there difference in the paths of the 2 files? Are you opening a folder with symlinks on the path?

[jkhoogland]

Hi,
sorry to reopen, but I have been having this exact issue with random corrupting of files on saving with clang-format on.

OSX 10.15.3
VSC 1.47.2
cpptools 0.29.0
clang-format 10.0

It is very hard to nail down a reproducible example, but I have switched off all extensions but cpptools 0.29.0.
When I select format-on-save using clang format, at random times, the file gets corrupted on saving.

When I undo the changes, move the cursor a couple of lines, or unselect any code that might selected, and again try to save it sometimes save correctly. This is a bit trial-and-error.

I am sorry that I cannot be more specific, I have not found a good way to reproduce, but that clearly is something going wrong.

Regards,
Jiri

[giellamo]

@jkhoogland do you have links in your Path?

[jkhoogland]

No, I work in files in a repo, no soft links.

[sean-mcmanus]

@jkhoogland If anyone can find a set of reproducable steps, that would help debug/fix the document corruption issue. Most likely, the formatting itself is not the cause of the corruption, but some other operation previously which corrupts the internal document object that is the input to the formatter.

[jkhoogland]

right, I really wish I could find some reproducible, but have not yet found it. Will try to find it and let you know.