Design: Zero-copy pass ArrayBuffer from JS-land to WebAssembly-land

[alexhultman]

Hey,

From reading about the WebAssembly.Memory there really is no way to pass an ArrayBuffer zero-copy to C++ from JS.

Example:

Let's say a WebSocket in the browser triggers the onmessage with an ArrayBuffer of the message data. Now I want to read this data in C++. From all solutions I've seen you really need to:

1. allocate a new WebAssembly.Memory (or whatever allocation strategy you use)
2. copy the contents of ArrayBuffer with the message to the ArrayBuffer of the WebAssembly.Memory
3. call into C++ and use the copied memory as char *
4. free the WebAssembly.Memory

Imagine having a constructor like new WebAssembly.Memory(ArrayBuffer) so that you could skip all steps except for step 3.

Did I get something wrong or is this not supported?

[lukewagner]

You're right that many Web APIs now incur extra copying overhead that we should work to remove.

The problem with what you've proposed is that WebAssembly engines often want to have special representations (extra guard pages, page-aligned allocation) for Memory which means you can't take an arbitrary ArrayBuffer and, without making a copy, convert it into a Memory.

I think a better way to achieve zero-copy is if, instead of allocating new ArrayBuffers as return values, Web APIs provided functions that took typed array views (or, to avoid view garbage, an ArrayBuffer, offset, and length) as input parameters into which the outputs were written.

[alexhultman]

I think a better way to achieve zero-copy is if, instead of allocating new ArrayBuffers as return values, Web APIs provided functions that took typed array views (or, to avoid view garbage, an ArrayBuffer, offset, and length) as input parameters into which the outputs were written.

Hmm, I'm not even sure this could be done in all cases. You don't know the size of a message up front and this sounds a lot like solving this problem of zero-copying by enforcing a completely new way of dealing with (in this case) networking. It is like killing a fly with a bazooka.

I don't really see this as a solution. For once it would require quite the extensive overhaul of many Web APIs and secondly, like mentioned, I think this is not even possible in all cases.

It seems super strange that something so essential to data management in JS, ArrayBuffer, would be so foreign to WebAssembly.

[alexhultman]

In the case of WebSockets you could maybe just add a new binaryType called "memory":

Attribute | Type | Description
-- | -- | --
binaryType | DOMString | A string indicating the type of binary data being transmitted by the connection. This should be either "blob" if DOM Blob objects are being used or "arraybuffer" if ArrayBuffer objects are being used.

This way all data passed to onmessage would be a WebAssembly.Memory.

[lukewagner]

For streaming cases like reading data from a socket, I think the place to do this is part of the Streams API. As Streams become more prevalent, this means more APIs would automatically become wasm-friendly.

The other big interesting case which isn't covered by streams is canvas/audio/video. I think these would/could know the size ahead of time.

Note that these changes would also be an improvement for JS: by allowing the caller to reuse an ArrayBuffer, it would produce far less garbage.

[binji]

I agree with @alexhultman, I think this is something we should fix in WebAssembly rather than requesting various web APIs to provide a view. Another problem with that solution is that if that API is provided, it's likely that the API will have to copy out anyway, since it likely can't trust the lifetime of the data in the view. Providing a mechanism for WebAssembly to access data from an arbitrary ArrayBuffer is a much more powerful and (IMO) useful feature.

It seems as though this is very naturally represented with multiple memories, which we've been planning from the start. This would mean that this WebAssembly.Memory is special, but I think it just means that you have to generate bounds-checked memory accesses, same as if you weren't able to allocate virtual address space for trap-handled memory.

There are definitely are some complications: the memory may no longer be page-sized, you'll likely want to unbind and rebind different memories, you can't use grow_memory, etc. None of these seem like show-stoppers, though.

[lukewagner]

Another problem with that solution is that if that API is provided, it's likely that the API will have to copy out anyway, since it likely can't trust the lifetime of the data in the view.

If the API is currently producing an ArrayBuffer, then there's already an inherent copy going on, so changing the API to take a view to write into is just replacing that copy. E.g., if you look at the Streams issue, the copy is async (b/c the given view is a view of a SAB and thus allows racy writes) and can thus be the direct recipient of the socket read syscall.

It seems as though this is very naturally represented with multiple memories, which we've been planning from the start.

While there may be special cases where only a single ArrayBuffer is to be read, most of the interesting use cases I've seen would process a new ArrayBuffer every {packet, frame, picture, ...} which would require creating a new wasm instance each time. So I don't think being able to import an ArrayBuffer as a memory is a general solution here.

Other problems with using an ArrayBuffer of input data as a wasm Memory include:

1. multiple memories don't translate naturally into C++/Rust b/c every load/store must statically refer to the right memory; that makes it hard to, e.g., take your bulk image data and just pass a pointer to libjpeg (which just uses the default memory for all loads/stores)
2. if attempting to use the ArrayBuffer as the only Memory (to avoid multi-Memory issues), there's no simple way to allocate some extra space for malloc, stacks, and other one-off allocations you might need for computation (without copying into a resizable Memory).

However, there is another persistent idea I should've mentioned above that could actually fit in as part of Host Bindings (if go the anyref route) which is to have a first-class slice value type whose values are basically typed array views. With this, load/store ops could be extended (via the flags immediate) to accept dynamic slice values (instead of static memory indices). This still has the downside of multiple memories described above, so I think the enhanced API route is still best, but it's a powerful raw primitive that could exposed to C++ in various ways.

[binji]

If the API is currently producing an ArrayBuffer, then there's already an inherent copy going on

Yes, I suppose so. Though it's possible that the API is providing direct access to its underlying data, where that can't be true with views. But you're right, maybe it's just changing who does the copy.

which would require creating a new wasm instance each time

I was thinking you could rebind the memory in that case, where the compiled code wouldn't bake in the address or size. Rebinding could be a similar operation to grow_memory in that case.

take your bulk image data and just pass a pointer to libjpeg

True, but perhaps it wouldn't be too much work to modify libjpeg to use the memory region pointers here instead. I don't know much about this C++ extension though (can't remember the actual name of it either).

With this, load/store ops could be extended (via the flags immediate) to accept dynamic slice values

I like this idea, but it does seem like it is more complicated than using static memory indices.

[lukewagner]

Though it's possible that the API is providing direct access to its underlying data,

The one example I can think of currently is where we create an ArrayBuffer whose backing store is a COW-mapped file. That case is definitely worth thinking about more since memory mapped file i/o is great. But other than COW mappings, since ArrayBuffers are mutable by JS once returned, I think it's almost always necessary to return a copy.

I was thinking you could rebind the memory in that case

I think 'rebinding' would need to be a new fundamental semantic operation, then? It's certainly implementable, but I think a bit weird spec-wise given that imports/definitions aren't defined to be mutable locations (that's what we have globals for; you could imagine having storing a reference to a slice in a global...).

True, but perhaps it wouldn't be too much work to modify libjpeg to use the memory region pointers here instead.

It's possible, but I think it's a fairly non-trivial change to make to an existing codebase in general. You have to put a special __attribute__ on every pointer statically indicating that pointer points into a non-default addrspace. This is hard b/c these pointers would be mixed with uses of the default linear-memory (stack, malloc, alloca etc) and in some cases a single pointer could point to either dynamically.

I like this idea, but it does seem like it is more complicated than using static memory indices.

Yeah, both more expressive and more complicated. If we're adding anyref and other opaque values (say exception) that require GC stack scanning anyway, the additional implementation complexity for a first-class slice could be modest, though, which is why I bring it up in relation to Host Bindings.

[alexhultman]

If the solution is a significant overhaul of the entire (JS) Web API then why not cut to the chase and just define a standard C Web API and cut JS completely out of the picture. Then you could reach and control the browser without the need for intermediate JS wrappers just acting as inefficient delegates.

[binji]

since ArrayBuffers are mutable by JS once returned, I think it's almost always necessary to return a copy

Right, unless the ArrayBuffer was detached, to represent transfer. But no Web APIs do that aside from postMessage, I suppose.

I think 'rebinding' would need to be a new fundamental semantic operation, then?

Yeah, I'm not entirely certain how it would work. But if we only allow it for linear memory, and a Memory object is just a pointer to its data and its length, then I think it has similar behavior to growing memory -- the data pointer and length are different. Unbinding would be like detaching the buffer; set data pointer to null and length to 0.

Ah, addrspace, thanks! Yeah, you're probably right that it wouldn't be trivial to modify an existing codebase to support that. But I'm guessing the compiler would help you here, so who knows? And you can probably assume aligned pointers and stuff the addrspace in the low bits. Or to be more careful, assume < 4G of memory and stuff the addrspace in the high bits, so access will trap. Then it looks you can use addrspacecast to turn it back into the correct addrspace pointer.

Yeah, both more expressive and more complicated.

I'm not sure how it solves the C++/rust problem though. You'll still need special objects to access this memory.

just define a standard C Web API and cut JS completely out of the picture

I assume you mean C bindings to existing web APIs? I like this idea, and I believe we've already talked about something like this for APIs like WebGL where there is an underlying C API that is well known. I'm not certain this can work for all Web APIs, though, as they often rely on JS-specific features that would be difficult (or maybe impossible) to provide from wasm.

[lukewagner]

I'm not sure how it solves the C++/rust problem though. You'll still need special objects to access this memory.

Right, that's what I said in my initial comment on slice; that it has the same problem as multiple memories.

If the solution is a significant overhaul of the entire (JS) Web API

I don't think "overhaul" is quite right. We're talking about adding overloads to some existing methods, in a way that can be done incrementally, for the hottest methods first.

[binji]

Right, that's what I said in my initial comment on slice

Sorry, misread that comment.

We're talking about adding overloads to some existing methods, in a way that can be done incrementally, for the hottest methods first

We should definitely do this, and I agree that it shouldn't be too much burden for most APIs. Like you say, they're probably doing a copy anyway.

My concern is mostly for cases where we aren't just giving data back to a web API, for decompressors and decoders and so on. We can just use typed arrays over WebAssembly memory for this too, but it's pretty unsatisfying to have to manage the lifetime of that data all the way through to the wasm module.

[lukewagner]

We can just use typed arrays over WebAssembly memory for this too, but it's pretty unsatisfying to have to manage the lifetime of that data all the way through to the wasm module.

I may not understand your meaning here but, except for cases that detach [[1](https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-decodeaudiodata)][[2](https://streams.spec.whatwg.org/#transfer-array-buffer)], Web APIs that want to use a view's data after the call returns need to make a synchronous copy, so it doesn't seem like lifetime would be an issue here.

[binji]

Here's an example of what I was thinking. Imagine you are using a zlib wasm module library. You don't know what the user of the library is going to do with the decompressed data, so you want to hand back an ArrayBuffer. This requires a copy-out from linear memory. Instead, you could hand back a Uint8Array view into the wasm memory, but then you need a way to manage the lifetime of the memory in the wasm module.

[lukewagner]

Ah hah, I see the direction you're talking about now. Yeah, I totally agree we should try to avoid such opportunities for wasm-level "leaks" and "use after free". So for the case of compression, encryption, and any large data processing, it seems like the best interface would be for the compressor to take a stream in and a stream out. This has parallelism and composability wins. I think we should strive, at both the toolchain and host-bindings-feature level, to make it easy/efficient to work with streams.

[briantbutton]

My vote for this feature, if I understand it correctly.

For general interest: There is a '.set' function for typed arrays which seems perfect to fill this hole.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/set

[jakirkham]

Sorry I'm a newbie to WebAssembly so am still learning and I may not fully understand the suggestions here. That said, am not sure streams are sufficient for solving the problem. They may make sense in the case of compression, encryption, etc. However things like FFTs generally require the full dataset to operate on or are at least difficult to implement for cases where only chunks of the data are supplied. So a stream won't cut it there. If the data is sufficiently large, copying the data will (if you are lucky) just be slow or (if not) crash the browser. What would you propose in this case?

[lukewagner]

The comments before the streaming comment discuss extending Web APIs to, instead of returning data in new ArrayBuffers, take caller-supplied views into existing ArrayBuffers where the data can be read into.

[jakirkham]

Views sound like a good idea. This is very similar to what people do to solve this problem in Python (e.g. the Python Buffer Protocol). Would it be possible to make read-only views? How would returning arrays work?

[lukewagner]

So far, JS/WebIDL doesn't have read-only typed array views. For "returning arrays", the general idea is to replace an array return value with a mutable view parameter that is written into. The hard part is picking the size of that mutable view argument: I think there's multiple options here, and probably different things would be appropriate for different APIs.