Vscode: VS Code fails to open big files (60MB)

[lllopo]

• VSCode Version: 1.1.1
• OS Version: Windows 10

Steps to Reproduce:

1. Try opening a, let's say, 60MB SQL file
2. VS Code says the file is either too large or in wrong format.

[alexdima]

We have made some improvements to the way we read files / create buffers. To investigate if we can raise the max file size limit.

[lllopo]

nope, no luck ... dies somehwere between 50 and 55Mb file size (still testing with UTF-8 sql dumps). I think you should reconsider the base of the file handling, so it does not depend on the file size at all (same goes for the lines length, btw. which are causing me troubles too).

[kirtangajjar]

Any updates on this one @alexandrudima

[oscar6echo]

Same here. VSC version 1.4.0 fails to open my 53MB csv data file.
I have to use Sublime while this is not solved.
It would be nice to unlock this limitation.

Besides this issue, I like VSC more and more. Congrats !

[lllopo]

@oscar6echo Yup, I must admit I also like VSC more, but sublime handles large files and long lines better, plus killer fast startup. So, If VSC needs urgent improvements, these are :

• large files handling - can't open at all
• long lines handling - can't handle very long lines at all, even with all wordwrapping and limitations off and syntax highlighters break with these too (probably due to not reading the whole line).
• serach is dead slow when average long lines (but withing VSC handling limits) present
• somewhat improve startup time

[zh-wowtv]

It's frozen when I load a 500K file. I'm using the latest VS code.

[alexdima]

@zh-wowtv Please create a separate issue and also attach the file that causes the freeze or a file similar in shape or nature that reproduces the freeze you're experiencing. I'm very interested in such cases, but each one is unique.

[faustinoaq]

I don't open huge files everyday.

But, when I do. I use neovim inside vscode terminal. _Simple and fast._

[pfmoore]

I hit this with a 130MB log file. Vim and Notepad++ open this without any problem, and searches it just as fast as a shorter file.

[anjia0532]

more than 50m,vs code throws The file will not be displayed in the editor because it is either binary, very large or uses an unsupported text encoding. and notepad++/sublime open it is easy! @alexandrudima

[anjia0532]

this i18n message in the https://github.com/Microsoft/vscode/blob/c67ef57cda90b5f28499646f7cc94e8dcc5b0586/src/vs/base/browser/ui/resourceviewer/resourceViewer.ts#L156

[anjia0532]

@alexandrudima
look this https://github.com/Microsoft/vscode/blob/39d761bf1f4ce7ea605539bdba41a8727b3c7ce0/src/vs/platform/files/common/files.ts#L484

export const MAX_FILE_SIZE = 50 * 1024 * 1024;

so vs code only open <= 50M files

open and replace this files

{VSCode Home}\resources\app\out\vs\code\electron-main\main.js
{VSCode Home}\resources\app\out\vs\workbench\electron-browser\workbench.main.js
{VSCode Home}\resources\app\out\vs\workbench\parts\git\node\gitApp.js
{VSCode Home}\resources\app\out\vs\workbench\services\files\node\watcher\unix\watcherApp.js
{VSCode Home}\resources\app\out\vs\workbench\services\search\node\searchApp.js

find number 52428800(50x1024x1024) modify to 52428800(500x1024x1024)

but i test the 223M log file , vs code is dead ,and i have to kill that .

open the same log file , vscode use more than 1.5G memory and nopad++ only 470M memory

[pfmoore]

What are the downsides of simply increasing that constant?

[anjia0532]

@pfmoore may be the big txt file make the system slow . so declare the MAX_FILE_SIZE ,i don't think it's a good idea

[pfmoore]

OK, so it does need better coding internal to VS Code to support this :-(

[alexdima]

The file size limit is somewhat arbitrary. Its sole purpose is to avoid OOM exceptions which lead to a crash. i.e. in our runtime (javascript) we cannot simply catch an OOM exception and stop loading a file, an OOM exception will crash the renderer process.

There are three possible "fixes":

• continue to improve and fine-tune in how we represent a file. This is already today highly fine-tuned, but we do represent each line as an object, and we do need a few pointers in each one of those objects (i.e. a pointer to the text on the line, a pointer to the tokenization state, a pointer to the tokens, a pointer to markers living on the line). We can of course work towards improving here. Please also keep in mind that we are in a VM. So when we are "logically" consuming 200MB, Chromium/v8 ends up oftentimes consuming much more real RAM.

• somehow compute up front how much memory opening a file would take. i.e. opening a 150MB file in a fresh instance of VSCode win 32 bit will probably succeed, but opening 20 50MB files will probably run out of memory. We would need to find out how much memory the process still has until it crashes and estimate how much memory opening the file would take. Then, we would refuse to open files until some other opened files are closed and some memory is freed.

• ship a 64 bit build on Windows. Under a 64 bit build, the renderer process will not crash at ~ 1.8GB memory usage (~1GB "logical" memory usage)

[pfmoore]

Thanks for the explanation.

continue to improve and fine-tune in how we represent a file

I imagine you've already done a pretty good job with this, so improvements here are probably minimal.

somehow compute up front how much memory opening a file would take

A better heuristic here would be good. I appreciate that opening 20 50MB files is going to hit memory issues, but that shouldn't mean that I can't open one. Would it not be possible to track usage at some level, so that if I open a 150MB file, it's "charged" to me the same as if I opened 3 50MB files? Then the limit could be expressed in terms of how much memory I'm actually using, rather than worst-case predictions. With the current arrangement, it sounds like I could still provoke a crash by opening many medium-sized files.

ship a 64 bit build on Windows

That would be useful. While it's all very well saying that opening a 150MB file shouldn't hit memory limits, the reality is that it does, and going 64-bit seems like a way round that. (Again, I presume it's not quite as simple as that, though, or you'd have done it already!)

But I'm sure you've thought through this sort of thing in great detail, If you're faced with the potential for the runtime to just crash on hitting memory limits, that's something you have to take into account (losing data in an editor is bad :disappointed:). At the end of the day, this is a code editor and 50MB files are almost certainly not source code. But while I don't know about other people, I tend to use my editor for looking at all sorts of text files, so it gets stressed way out of its actual designed use case.

Hmm, one probably impractical thought - could there be a readonly mode for opening files that doesn't keep the file content in memory, as it's not going to be changed? That could be used by default for files over a given size. Of course having to read data from disk all the time might make things dreadfully slow, and the alternative data format would likely have massive effects on the rest of the code, which would have to be updated to work with the new format. I doubt this is a realistic option.

Anyway - sorry for rambling, and thanks for taking the time to explain the issues involved.

[anjia0532]

@alexandrudima Thanks for the explanation.

ship a 64 bit build on Windows. Under a 64 bit build, the renderer process will not crash at ~ 1.8GB memory usage (~1GB "logical" memory usage)

but wiki/How-to-Contribute#packaging seems only support win32 not win64?

[alexdima]

@anjia0532 Yes, please upvote #507 -- there is no technical limitation AFAIK

[lllopo]

64bit version is a good idea, but I'm still wodnering how SublimeText handles big files so easily.

[PoisonousJohn]

When we're talking about huge files, there's no need to process it whole. Typically only the part visible to user (+- page) is processed.

I'd prefer that syntax highlighting or other parts depending on tokenization be disabled, but file will still be displayed and operable.

Lack of this feature forces me to use other editors.

[lionello]

@alexandrudima Creating one object per line is fine, as long as you don't try to do it for the whole file at once. Perhaps doing 1000 lines at a time, and then read the next 1000 and releasing the last 1000 once the user jumps/scrolls there? (With some overlap.)

[benyaminl]

Any to do list for this? Is there anyone consider make js work on top of OS rather than on chromium hehehe...

[dinazil]

I'm on Windows 10. When I open a 27MB file it opens, but as soon as I try editing it, the window crashes.

[noma4i]

1. VSCode associates itself with *.sql files right after been installed.
2. Not able to open most of them because of file size.
3. ....

[MikeGitb]

I'd give you a laugh emotji, If it wasn't so sad ...

[benyaminl]

Applause for VS code :clap: :clap: :v:

[benyaminl]

@faustinoaq Then what the Vs code for :v: :laughing:

[MikeGitb]

I don't know anything about the inner workings of vscode, but the solution mentioned by @IvanFateev and @lionello seems most logical for me:
If the file reaches a certain size, go into a fallback mode, where not all the file is held in memory at the same time. Of course, this might be a pretty complicated change, if the whole codebase currently assumes that it has access to the whole file at all times.

[olaeld]

Won't open a 51 939 KB file 😟

[alexdima]

Improvements have been made in #30180 , and our memory usage is now roughly:

file size on disk + 40-60 bytes (2 objects) per line + an array of pointers of length line count

We will look into removing or raising the limit on 64 bit platforms

[isidorn]

The size has been raised for 64bit vscode on windows. You can try out the 64bit insiders version or wait until we release stable in the following week.
I tried opening a 77 mb csv file and everything worked fine -> verified.

[lllopo]

I can confirm. Tried the 64bit insiders with a 177MB SQL file - opened and opened reasonably fast.
Scrolling not perfect - initially the scroll was not reflecting the real length of file, but overall - works just fine. Also no syntax highlighting, but that has to be expected, I think.

[anjia0532]

how to download 64bit? build by myself?

[benyaminl]

No just download from the official download page select the 64bit version.
Done.

Benyamin Limanto

sent from my ASUS FonePad 8

Pada tanggal 3 Agt 2017 15.22, "anjia0532" [e-mail]
menulis:

how to download 64bit? build by myself?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/6474#issuecomment-319902241,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NEOdARpktdohRW3_2zZN_-foUay8ks5sUYNBgaJpZM4Ig_ra
.

[anjia0532]

@benyaminl thanks for your reply and I tried opening a 283 mb json file . in this demo vscode use 6 process and more than 500M memory.

the nopad++ not open this file.

sublime text 3 build 3126 use 1 process and more than 700M memory.

[benyaminl]

Errr. The 64bit now ha the stable version. I think you should check it.
Hehehe...

Benyamin Limanto

sent from my ASUS FonePad 8

Pada tanggal 3 Agt 2017 23.19, "anjia0532" [e-mail]
menulis:

@benyaminl https://github.com/benyaminl thanks for your reply and I tried
opening a 283 mb json file . in this demo vscode use 6 process and more
than 500M memory.

[image: vscode]
https://user-images.githubusercontent.com/15098916/28931715-888fb086-78a9-11e7-88fb-0594d1978884.png

the nopad++ not open this file.
[image: nodepad]
https://user-images.githubusercontent.com/15098916/28931850-14adbfd6-78aa-11e7-95b7-dcb710b15580.png

sublime text 3 build 3126 use 1 process and more than 700M memory.

[image: sublime]
https://user-images.githubusercontent.com/15098916/28931971-810d8044-78aa-11e7-8b7f-9e3446fc8dfa.png

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/6474#issuecomment-320018560,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NKvi9QmuZmk3EIpR0lq_Lw0Yi1Opks5sUfMngaJpZM4Ig_ra
.

[anjia0532]

if i use the stable vscode 64bit it's failed to read this json file.

but insider vscode 64bit can be read

[benyaminl]

Okay, it seems they still working on it.. Then Ok. Just wait for the new version released, Great!

Benyamin Limanto

sent from Windows Live Mail on Windows 7 (Asus A43SV)

From: anjia0532
Sent: Friday, August 04, 2017 8:10 AM
To: Microsoft/vscode
Cc: Benyamin Limanto ; Mention
Subject: Re: [Microsoft/vscode] VS Code fails to open big files (60MB) (#6474)

if i use the stable vscode 64bit it's failed to read this json file.

but insider vscode 64bit can be read

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.