Node: Remove 1GB max size limit for Buffer

[feross]

Buffer instances currently have a max size of 1GB (enforced in smalloc.h). This isn't ideal.

I know Buffer is being rewritten for V8 4.4, but wanted to open a separate issue to track this limitation.

[petkaantonov]

The limit comes from v8 implementation details, there is nothing we could do about it in node.js

[bnoordhuis]

What @petkaantonov said, it's because of this V8 limit, which is in place for a number of reasons, mostly to do with V8's use of tagged integers (or SMIs) to store the length.

The switch to typed arrays in 4.4 isn't really going to change that because the same size restriction applies to typed arrays (and the arraybuffers that back them.)

I'll close the issue. It's something that is outside of our control.

[feross]

Fair enough. For anyone interested, here is the V8 bug to star for this issue: https://code.google.com/p/v8/issues/detail?id=3505

[petkaantonov]

Actually that issue is not related to buffer sizes but other objects that currently have even lower limits than buffers, for different reason.

[feross]

Don't typed arrays subclass from Array now?

[petkaantonov]

They don't subclass array but that is not related to these issues. The issue you linked is about increasing string and regular array sizes from already low limits (e.g. 128mb or 512mb) to the maximum possible smi limit (e.g. 1GB). The smi limit itself is a separate issue that cannot be raised (well you could make it 2GB on 64 bit).

[feross]

Sounds like the 1GB limit will be lifted with the move to V8 4.4 after all. See: https://twitter.com/trevnorris/status/603345087028793345

[trevnorris]

@feross ArrayBuffer's can be arbitrarily large. It's that the index size cannot exceed Smi::kMaxValue. So while a Uint8Array can only address a ~2GB range on x64, a Uint32Array could cover ~8GB. (fyi, my ticket to remove the Typed Array limit: https://code.google.com/p/v8/issues/detail?id=4153)

@domenic has suggested we allow ArrayBuffer to be passed to all APIs that support an Buffer. I'm considering how that could be done to give us a sweet spot of allowing the transfer of data within node to be encapsulated in an ArrayBuffer (support for arbitrary size) while also allowing to work with the data using multiple buffer instances if needed.

[feross]

@trevnorris For starters, we could just make the Buffer constructor take an ArrayBuffer (https://github.com/nodejs/io.js/issues/106) and work our way towards @domenic's suggestion.

[trevnorris]

@feross We'd have to also allow Buffer(arraybuffer[, start[, end]]) if we want to support making Buffer slices of an ArrayBuffer.

[oleersoy]

I think I'm running into some of the consequences of these limits here:

https://stackoverflow.com/questions/44959025/rangeerror-maximum-call-stack-size-exceeded-caused-by-array-splice-apply

Surprised that I'm getting a call stack exception though....

[bmeurer]

FYI: I'm working on supporting big TypedArrays and DataViews in V8 right now. For 64-bit platforms we should be able to support buffers way beyond 4GiB soon (32-bit will probably remain limited to a little less than 2GiB, but usually you'll struggle to find this much contiguous virtual memory anyways).

[jt0dd]

@bmeurer did this ever happen? https://stackoverflow.com/questions/64056286/in-2018-a-tech-lead-at-google-said-they-were-working-to-support-buffers-way-bey

I don't understand how 64 bit range buffer addressing is beyond the reach of Node on 64 bit systems in 2020

Edit: If I understand right, it's actually a solved issue in V8, but Node.js still decided on a 4294967295 buffer limit for some reason

[bnoordhuis]

@jt0dd require('buffer').kMaxLength reflects v8::TypedArray::kMaxLength, which is still 2**32-1 on 64 bits architectures:

https://github.com/nodejs/node/blob/785a5f9ae1674118ffeae4525080c6d3ede8fd3e/deps/v8/include/v8.h#L5348-L5355

As to that SO post, see commit 5005c3c72c232915935b97800781467767964660: most platforms don't allow I/O operations > 2 GB.