Web-audio-api: AudioContext.decodeAudioData() from a SharedArrayBuffer source?

[juj]

Multithreaded WebAssembly applications utilize a SharedArrayBuffer object as their heap type. These applications can contain encoded audio files inside their heaps, e.g. as part of a filesystem that they embed inside the heap.

However, these applications are unable to decode audio files from within their WebAssembly heaps. Calling

var sab = new SharedArrayBuffer(16);
var ac = new AudioContext();
ac.decodeAudioData(sab);

fails with

TypeError: Argument 1 of BaseAudioContext.decodeAudioData does not implement interface ArrayBuffer.

As a workaround, the applications need to create a temporary copy of the data from a SharedArrayBuffer to a regular ArrayBuffer, e.g.

var sab = new SharedArrayBuffer(16);
var ac = new AudioContext();

var ptr = /*address to start of .ogg file=*/0;
var length = /*length of .ogg file=*/16;
var audioData = new ArrayBuffer(length);
new Uint8Array(audioData).set(new Uint8Array(sab).subarray(ptr, ptr+length));

ac.decodeAudioData(sab);

If Web Audio spec allowed decodeAudioData directly from SABs, then the above deep copy of encoded .ogg files would be avoided in multithreaded WebAssembly applications.

CC @lars-t-hansen

[lars-t-hansen]

This is mostly a specification wrinkle + some WebIDL, I expect; @annevk? From the point of view of the implementation it needs to make sure to use safe-for-races accesses (to avoid C++ UB) where that matters. I see @padenot is paying attention already so not going to do anything here myself.

[padenot]

This is something that would be very nice to have indeed, however it's not without problems.

The main issue I see is that decodeAudioData has to be the sole owner of the memory region on which it operates, for obvious safety reasons. In short, internally, we send this buffer to another browser subsystem, which does the decoding. This can in turn use various ways of doing the decoding: some implementation send it to ffmpeg, some use the system's facilities, some use a library to do the decoding. On mobile, it's not uncommon to have some hardware helping with the decoding process (most often for mp3 and AAC, iirc). This all depend on the codec, implementation and platform.

Having the possibility to mutate the bytes while the decoder is working probably cannot be proved safe in practice (due to the multiplicity of scenarios, a lot of which browser vendors don't have any control on). It used to be that the buffer wasn't detached, and this caused more than one security issue, iirc.

I'm particularly concerned by codecs that try to keep the memory usage down and locality high by not copying information out of the container, and instead referencing bits of memory. Let's say you get a length out of the container, at a certain offset, validated initially, and then a malicious bit of code changes the length under the codec, that still references this memory, and this can potentially cause an OOB. I can't prove that this is something that is not done today in code that we don't control, and I can't prove that this is something that cannot be done in the future.

In addition (and this is an implementation concern, not a spec concern), Firefox is now running the codecs in a separate process (because they are very complex piece of code, we're sandboxing them harder than the rest of the web content). This means that we would copy anyways (either to a SHMEM, or via a pipe, etc.), to decode. However this does it by block by block so the copies are smaller and the memory usage is not doubled.

Is there a facility in SharedArrayBuffer to mark a range as read-only for WASM for a period of time, with a lifetime for this read-only period dictated by the implementation? This would solve our issue nicely here, if all we want is to save the copy (and I presume saving a pair of allocation/deallocation, and having a transient higher memory usage).

Another thing that could be interesting would be to decode _into_ a SharedArrayBuffer. That's probably easier, and I think it would be quite important, PCM data is often quite big, and authors using WASM would be happy to skip a copy I'm sure.

[lars-t-hansen]

Is there a facility in SharedArrayBuffer to mark a range as read-only for WASM for a period of time, with a lifetime for this read-only period dictated by the implementation?

Not at this time. We're talking about adding mprotect-like features to wasm in the future but it's unknown how that would interact with the needs of the audio subsystem -- I would think it might just make things even more complicated.

[annevk]

From an IDL perspective, you'll need to use [AllowShared] for starters. There's then also the question as to whether the SharedArrayBuffer object can be used to create a high-resolution timer, assuming postMessage() functionality for it was disabled.

That is, we want to enable SharedArrayBuffer everywhere, but guard any usage of it that can create a high-resolution timer. If your feature can be used to create a high-resolution timer independent of other features (in particular including the sharing it with other threads one), you'll need an additional opt-in.

Does that make sense?

[hoch]

Is there a facility in SharedArrayBuffer to mark a range as read-only for WASM for a period of time, with a lifetime for this read-only period dictated by the implementation?

For the same reason, it would be great if we can use SAB for Audio Worklet's input and output buffer. The mechanism @padenot described above will be really helpful in any use case where the system (renderer, decoder and etc) needs to claim the exclusive access to a given memory space (or range), then releases the ownership when the task is completed.

[lars-t-hansen]

I don't really see that kind of mechanism being possible until wasm has something like mprotect and exception handling and we can phrase the semantics of the protection in those terms. In that case, we may be able to treat the audio subsystem simply as a concurrent agent that protects/unprotects parts of shared memory, and we'll have a model for what it means when other agents touch the protected area.

[padenot]

I think it's best to wait until wasm/SAB has the facilities needed here.

@lars-t-hansen, if discussions happens on something like mprotect/etc., can one of us be notified? I don't think we can do anything without it.

[hoch]

we may be able to treat the audio subsystem simply as a concurrent agent that protects/unprotects parts of shared memory

@lars-t-hansen I really like that idea. This new design will help a lot of performance-critical audio use cases.

We're punting this to v.next, but Audio WG might not be the right venue for discussion. Web Audio API or the other audio API will be a customer of such design.

[juj]

Solving this at a higher level, if #1305 happened, and a more efficient cursor-based decoding API was available, then that could be used as a replacement. In such scenario, many applications would be able to avoid having to stuff the compressed audio files inside the wasm heap in the first place, if the browser was able to decode audio for them.

[mdjp]

Closing this, it is not an area of the api that we will be working on in V2. However we hope that this
will be covered by https://github.com/WICG/web-codecs
https://github.com/WICG/web-codecs/blob/master/explainer.md
https://discourse.wicg.io/t/webcodecs-proposal/3662

[padenot]

(It's a requirement of the web-codec effort to work well with the web audio api).

[guest271314]

Just tried this yesterday. Would be useful.

[hoch]

It's a requirement of the web-codec effort to work well with the web audio api

Probably it's better to get that in the WebCodec explainer. :)

[guest271314]

@hoch Also, to clarify precisely what "work well" means. "work well" to do exactly what?

[hoch]

Oh, I was merely quoting @padenot's comment - but "working well" means many things in many different layers; performance, API ergonomics, and ease of use. I won't try to clarify everything in this comment or this thread.

[guest271314]

@hoch Have read several times in this repository that issues are closed referencing "WebCodecs" as some prospective future solution to a current Web Audio API limitation. If Web Audio API is openly relying on an as-yet un-specified and un-deployed API it would be helpful to post - somewhere conspicuous - exactly how that "working well" is expected to be observed. There is no evidence to support such claims right now.

Also, if this repository is no longer accepting feature requests can you kindly remove that language and option from your issue template?

[rtoy]

The feature request template is being removed. We'll probably need to add some words somewhere that feature requests should go to the issues for the v2 repo.

[guest271314]

@rtoy For example, precisely how precisely will WebCodecs solve the problem of decodeAudioData() and decoding audio that either does not begin at the start of the file or is capable of prepending a required header to the media fragment https://github.com/WICG/web-codecs/issues/28? That use case is nowhere close to being solved by WebCodecs. Meaning if Web Audio API is relying on or deferring to WebCodecs to "work well" with Web Audio API, that can be an open-ended and indeterminate waiting period for a solution with no clear timeline or responsible party. Who is the lead in such case, WebCodecs or Web Audio API? Slogans such as "work well" are inconclusive and provide no definitive information whatsoever. Kindly write out the agreed upon "working well" flow chart, if one exists at all, other than the thus far used response "Web Codecs" is going to solve that. Am not convinced at all.

[rtoy]

Don't know how it will work, but having two apis that overlap a little is not a great pattern. It seems more beneficial for everyone if WebCodecs can solve this, rather than grafting some kind of partial solution to decodeAudioData but not be able to do other things that WebCodecs can do. It's up to you and us to do our best to make it happen.