Vscode: Save: Flush to disk after writing to file

[garfieldbanks]

• VSCode Version: 1.3.1
• OS Version: W10

Steps to Reproduce:

I don't have exact steps. It was working fine. I was writing node js scripts to split old family videos into individual clips. I was using the built in terminal. Midway through a script I canceled the process. After I fixed and restarted the script I noticed ffmpeg wasn't running. It was frozen or something so I rebooted my computer. It wouldn't restart so I held the power button until it turned off. On reboot I reopened vscode and I can no longer see javascript files:

The file cannot be displayed in the editor because it is either binary or uses an unsupported text encoding.

[garfieldbanks]

Opened file with notepad and it is blank.

Looks like vscode deleted the contents of my file.

I was really starting to like vscode. But this sucks.

[garfieldbanks]

I can open javascript files from old projects but all the *.js files in the folder I had open in vscode are blank and give the "file cannot be displayed..." message.

[kieferrm]

@garfieldbanks I'm sorry you lost your data. That's bad. However, I'm not yet ready to accept the blame... All your files where saved when you rebooted your box? Did you have autosave enabled or not?

[kieferrm]

See also my comment in #9552. The issue describes a similar experience also on W10.

[garfieldbanks]

No I don't have autosave on, but I don't think autosave is relevant as this happened to many of the files in the project, most of which were not being edited anywhere near the time of the freeze. There is only one file that I edited before restarting the script and that is the main script itself. But I saved it _before_ restarting the script. So none of my files should have been in an unsaved state.

I'm sorry that I was a little angry when I made this issue. I've since rewritten the code (it wasn't really all that much). I understand that these things happen and I still like vscode.

If there's anything else I can do to help figure out the bug, please let me know.

[benlowry]

I don't have auto-save enabled either and lost my pre-existing file when the power cut and I had unsaved changes. Windows 10 Pro, current VSCode release (1.3). The only setting I have altered is using \n as line endings.

[joaomoreno]

cc @emragins

[joaomoreno]

I can easily reproduce this with VS Code. But does it reproduce with another applications...?

An interesting experience would be to open VS Code in a workspace (which watches over the files within), use Notepad to edit a file inside that workspace and crash after the save operation. The theory is that the file watching would have some factor in this. After a few attemps, I could reproduce it, but in this case instead of the file being completely _zero filled_, only the newly added content would be _zero filled_ (I typed 7 bytes before the crash):

Hmmm... I didn't really believe file watching had anything to do with it so I tried reproducing it with Nodepad only. After many attempts, I did it:

While Notepad still has the issue, it does seem to behave a bit better, by limiting the damage to the newly added bytes. So this lead me to compare what Notepad does versus what Code does.

Notepad

Code

Well, both processes seem to write the full file to disk, but Notepad seems to cause two more events (SetEndOfFileInformationFile and SetAllocationInformationFile) to make sure the file is truncated at 1144 bytes... is that really making a difference here? The seem to come from calling SetEndOfFile from KERNELBASE.dll:

To be continued...

[joaomoreno]

Let's boil it down to a snippet:

const fs = require('fs');

fs.readFile('LICENSE.txt', 'utf8', (err, contents) => {
    if (err) contents = '';

    contents += 'hello world\r\n';

    const buffer = new Buffer(contents, 'utf8');
    fs.writeFile('LICENSE.txt', buffer, err => {
        if (err) return console.error(err);
        console.log('done');
    });
});

It wasn't hard to reproduce the issue with this code: after a few attempts, the whole file was filled with zeros.

So.. let's try to replicate Notepad's behaviour and truncate the file before closing it:

const fs = require('fs');

fs.readFile('LICENSE.txt', 'utf8', (err, contents) => {
    if (err) contents = '';

    contents += 'hello world\r\n';

    const buffer = new Buffer(contents, 'utf8');

    fs.open('LICENSE.txt', 'w', 0o666, (err, fd) => {
        if (err) return console.error(err);

        fs.write(fd, buffer, 0, buffer.length, err => {
            if (err) return console.error(err);

            fs.ftruncate(fd, buffer.length, err => {
                if (err) return console.error(err);

                fs.close(fd, err => {
                    if (err) return console.error(err);

                    console.log('done');
                });
            });
        });
    });
});

Here's the process events. It seems that at least the SetEndOfFileInformationFile event is fired. Not sure how to get the SetAllocationInformationFile.

But that wasn't enough, it seems, as the whole file was zeros, after reproducing the issue:

I decided to stop here, since I don't think we'll get close to Notepad's behaviour. Also, even if we did the issue would only be improved, not fixed.

It's possible that the _solution_ would lie on the SetAllocationInformationFile event, which I have no idea how to call from Node. Another theory is that the CreateFile call is slightly different as well, between Notepad and Node. I couldn't figure out a way to make the same call in Node as its done in Notepad.

Interesting link: http://stackoverflow.com/questions/12228042/what-does-setfilevaliddata-doing-what-is-the-difference-with-setendoffile

[joaomoreno]

I will close this issue as there's not much we can do here.

Mitigation suggestions

• use an SCM tool, such as git;
• use a backup system, such as Windows Backup and Restore.

Closing arguments

Above all else, note that a crash is an unexpected error which can hardly be always properly handled. Anything can fail (and often will) during a crash, from user-land apps, through the kernel down to actual transistors.

There are a few ideas of how we can mitigate this such as saving the file to another location and only then overwrite the original file; this way there would be copies of the edited files somewhere in disk. But even that can fail: the FS could batch the backup and save operations together and the machine could crash while applying those operations, thus losing all state...

It stands to reason that finding and fixing the source of the crash is the only sane fix; in this case this means charging your battery more often or simply not crash the machine on purpose.

[joaomoreno]

cc @Microsoft/vscode: thoughts?

[emragins]

Thank you for the time and effort to look into this, but I'm leery about closing this issue. As I indicated in #9730, I have no reason to believe that my laptop power was the course of my computer's crash since it was both plugged in and fully charged -- no weird overhead light flicker either. I also think it's interesting that this same error has been reported three times in the span of less than a week. If this current thread is the first known occurrence that seems like more than a coincidence to me.

Were the side-effects of the problem less devastating perhaps I wouldn't mind the close, but I'm honestly a little paranoid to continue using VSCode, even though I've grown to really like it as my primary js editor.

[emragins]

Wow, sorry for the spam -- my phone did not reload or indicate that I clicked "comment". Deleted all the duplicates.

[joaomoreno]

@emragins I agree with you.

It is rather odd that there were 3 occurrences in the timespan of a whole week. But given that I can reproduce this with other applications (such as Notepad.exe), this doesn't appear to be a new issue. I'd place my bets on the fact that we always had it; and that Windows just decided to crash 3 separate times on 3 separate users in a week time span... I too am sceptical about coincidences, but I believe this was one.

I didn't mean to say that low battery was your specific problem. But your machine did crash down, effectively making your issue a duplicate of @garfieldbanks's and @benlowry's.

It is unfortunate that this makes you trust Code less but, as shown in the comments above, other (if not all) applications will have the same issue.

I've been trying to reproduce this with Sublime Text and I seem to fail at doing so. When using Code or Notepad, if I crash the machine too soon after the save, the file doesn't even appear touched after reboot. Using Sublime Text, I can't seem to be fast enough to get this to reproduce. The file is always modified and has the correct contents. Meaning: they must be doing something smarter than us. Let's investigate why that is...

PS: no worries about the spam 😉

[joaomoreno]

Here's what Sublime is doing:

That FlushBuffersFile operation is highly suspicious. 😉

[joaomoreno]

I'll put this to August and take a closer look at how we can improve here with @bpasero.

[emragins]

Awesome. Thanks! :)

[benlowry]

I share all of @emragins' concerns, and the additional concern that a bug in Sublime is somehow justification to write this file-destroying issue off, seems far more like "they have to fix it too".

[garfieldbanks]

I don't think @joaomoreno is writing it off. I believe his last comment meant that he was going to be enlisting @bpasero to help him figure it out and that he expects to have it done sometime in August.

[joaomoreno]

@benlowry The fact that other applications have the same problem means the issue isn't with the applications, but with the OS, file system, kernel, hardware, or something down below.

But the fact that Sublime doesn't seem to have it, by using the FlushBuffers operation, means we can at least be as good as them. This doesn't mean the problem won't exist, we're just going to narrow down the time window in which is it possible to occur. A crash is always a crash, and very little can be done about a crash at the _right time_.

[bpasero]

@joaomoreno I pushed this and would appreciate a verification on Windows with your setup.

[joaomoreno]

This seems to work beautifully.

[bpasero]

@joaomoreno thanks for verifying <3