Node-fs-extra: Error: Cannot create a string longer than 0x3fffffe7 characters

[tommedema]

• Operating System:
  macOS Mojave 10.14.1 (18B75)

• Node.js version:
  v10.14.2

• fs-extra version:
  7.0.1

I'm running this code on a text file that is 1.13GB large:

const csvString = await fs.readFile(absFilePath, { encoding: 'utf8' })

Unfortunately it exits with:

Error: Cannot create a string longer than 0x3fffffe7 characters
    at stringSlice (buffer.js:594:43)
    at Buffer.toString (buffer.js:667:10)
    at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:48:23)

Since the function is asynchronous I wouldn't expect this. The library should take care of buffering etc. for me and return the result once done.

[RyanZim]

You're dealing with a file that is far to large to load into memory at once. We cannot buffer the file for you, etc. because the size of the file is larger than Node.js itself is capable of creating a string for; actually, larger than Node.js is able to store in memory at once as well. You're going to have to use streams to work with such a big file.

fs-extra does nothing to fs.readFile except convert callbacks to promises.