Web-audio-api: Enable AudioContext to be created in a Worker

Created on 11 Sep 2013 · 20Comments · Source: WebAudio/web-audio-api

Originally reported on W3C Bugzilla ISSUE-19991 Sat, 17 Nov 2012 17:12:35 GMT
Reported by Jussi Kalliokoski
Assigned to

Currently there are several annoyances to scheduling in the Web Audio API. For example, if you want to play back a dynamic sequence, you would power it up with a setTimeout/setInterval, both of which are throttled to once per second when in a background tab. Now, if you set up a timer that happens once per second what happens if a reflow or other event delays the timer? To protect against this, you can schedule events for more than a second at a time, but it's a tradeoff for responsiveness of the application. Responsiveness or robustness is not a nice tradeoff to make.

A suggested approach to this problem has been to add a callbackAtTime() method to the AudioContext, but I fear that introducing yet another timer mechanism to the main thread won't help much. Say you setup a callback to trigger one second from the current time. Should it

a) Fire before the clock actually hits the specified time to be a bit more sure to make it in time?
b) Fire exactly when the clock actually hits the specified time? In this case, the desired target is most likely missed.
c) Fire after the time? This is a ridiculous idea. :D

Anyway, even that would be suspect to being delayed by other main thread events like reflows etc., being not much more reliable than a setTimeout(). I think we're going to get a lot of "no" sound from other working groups and browser vendors if the callbackAtTime() had no throttling rules, when browsers finally have painfully put those restrictions to place for existing main thread timer callbacks, so I don't think we'd get even that advantage.

Hence I'd suggest specifying access to the AudioContext interface from Web Workers, where one doesn't need to worry about main thread events delaying anything, nor about timer throttling.

For the time being, the Workers would obviously support less features (supporting MediaStreamSourceNode and MediaElementSourceNode in the Workers would require transferring these entities to the Worker as well). One option would of course be that AudioContexts would be defined as Transferrables, as well as a AudioNodes, letting graphs be shared across threads. This would probably actually be the best way to achieve this, provided we can eliminate race conditions by having value setters and getters exclusive to the thread that currently has the ownership of each node. But there aren't many critical features like this in the Web Audio API, which makes it a prime candidate for being a Transferrable.

Feature RequesMissing Feature

Source

olivierthereaux

👍7 ❤2

Most helpful comment

+1
Actually running an AudioContext inside the main thread is never a good idea apart from a small demo or example.

andremichelle on 24 Mar 2014

👍3

All 20 comments

I'm currently using a script processor to schedule audio events 1024 samples ahead for sequencing. This works well until the UI does something crazy like draw a waveform or some sort of reflow causing a bit of jitter.

Making AudioContexts transferrable and/or exposing AudioContext in WebWorkers would solve all of my problems!

mmckegg on 3 Jan 2014

+1
Actually running an AudioContext inside the main thread is never a good idea apart from a small demo or example.

andremichelle on 24 Mar 2014

👍3

Well, in Chrome at least, the actual audio processing is not happening in the main thread anyway (aside from ScriptProcessors, which is a separate issue). I still think this ability would be useful - for example, to run your scheduler in a background thread - but you should be able to create scriptprocessors as workers anyway, and it should absolutely not be true that "running an audio context inside the main thread is a never a good idea" - that may be a poor design choice in some implementations, but audio processing shouldn't be affected by long main thread events.

Transferable context and other audio objects may complexify this, of course.

cwilso on 24 Mar 2014

Maybe I didn't found the right solution, yet. All I need is to be able to compute audio in a worker by JS and maintaining a glitch-free playback even the UI-thread has a very high load for some seconds (worst case scenario). Any hints?

andremichelle on 24 Mar 2014

You need something that isn't there yet - captured by issue #113 . You don't need the entire AudioContext in the worker, just the ScriptProcessor's processing loop, was what I meant.

cwilso on 24 Mar 2014

Exactly. Anything I can do to up-prioritise a solution? I am working on a html5 version of www.audiotool.com, but it is already clear that we cannot compete against the current flash-version, if this issue is still unsolved. Thanks for your work on audio. Much appreciated!

andremichelle on 24 Mar 2014

Do my job for me on that issue and write up a proposal. :) Seriously, though, I do see this as a serious issue, and it's blocked next on me writing up a proposal and getting agreement on it.

cwilso on 24 Mar 2014

Yes, serious it is! Let me know, if you want me to spam some guys with proposals ;)

andremichelle on 24 Mar 2014

No spam necessary - just write up a proposal and put it in the issue.

On Mon, Mar 24, 2014 at 11:27 AM, andremichelle [email protected]:

Yes, serious it is! Let me know, if you want me to spam some guys with
proposals ;)

Reply to this email directly or view it on GitHubhttps://github.com/WebAudio/web-audio-api/issues/16#issuecomment-38482295
.

cwilso on 24 Mar 2014

Just to be clear: What kind of proposal are you talking about? I do not propose a change in the API itself. It should only be possible to create the audio-callback in a worker.

andremichelle on 24 Mar 2014

That is, in fact, a change in the API - instead of handing attaching an onaudioprocess event handler to an object in the main thread's scope, you'll have to fork off a process with a creation mechanism that accepts an URL with the handler, similar to how Web Workers are forked normally.

That doesn't, of course, necessarily change how the audio event process happens, or how onaudioprocess works; other than that it would need two objects - the ScriptProcessorNode-like object in the main thread that is used for routing purposes, and the Worker's global context object, which is where the onaudioprocess event would actually be fired (and the code for that event would be in the URL-hosted resource, not attached directly to the main thread Node object as it is today).

In some cases, this can be minimized to some boilerplate, but it's important to understand the design of thread isolation is going to cause some necessary complexity; for example, any data or parameters will need to postMessage()'ed across from the main thread.

cwilso on 24 Mar 2014

https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-27095326, BTW, is a pretty good first stab at what the implementation would likely look like.

cwilso on 24 Mar 2014

I was actually wondering how I can additionally support this request without repeating what has already been suggested in https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-27095326

As a sidenote: In the Flashplayer I can create a Worker where the callback Event will be attached to a Sound Object, so that the entire audio chain is not leaving the Worker. You can even go into an endless loop in the Main-UI Thread and the Sound will continue playing. That is exactly what we need.

andremichelle on 25 Mar 2014

I'd encourage you to weigh in on that issue thread with use cases.

I will point out that any audio that you DON'T compute in a script processor - i.e. any web audio nodes that are processing audio - won't be horked* by the main thread being in an endless loop. The audio will always leave the Worker thread, because it has to be traded back and forth with the audio thread - and we can't just put your audio workers in the audio thread, because 1) that would enable JS developers to hork* the audio thread with arbitrary JS - even accidentally, as the audio thread would then have a JS garbage collector running in it - and 2) the audio thread is a high-priority thread, and we would not want to put JS in a high-pri thread.

yeah, hork. It's a thing. :)

cwilso on 25 Mar 2014

Any news on this issue?
I am axperimenting on audio DSP on browser, and it will be quite useful to be able to do this on a web worker instead of the main thread.
Just my +1 vote to be able to load AudioContext on a web worker.

leomrocha on 30 Jun 2014

Quoting from an email from Ashley Gullen [email protected]:

We develop a HTML5 game engine and our goal is to ultimately have the entire engine running from a Web Worker, off the main thread.

There is progress on getting canvas rendering from workers specced. The last major hurdle to running an entire game engine in a worker is audio playback. Currently AFAIK Web Audio is only specced to work from the main thread, and it is a relatively complex API, so if a game engine is running in a worker there would need to be a bridge between the worker and the main thread with lots of postMessage going on, which could impact performance and add latency while adding complexity. Having Web Audio available in Web Workers would solve this. Has this been investigated yet?

Ideally AudioContext could be made available in WorkerGlobalScope. However I can immediately see a number of issues:

integration with Media Elements would not be available (e.g. createMediaElementSource), because they are part of the DOM which is not available in a Worker
integration with Media Streams would not be available, including from getUserMedia, since these are not available in a Worker (AFAIK)
selection of audio input and output devices is currently only available on the main thread (although this may not be a problem if device ID strings can be posted to the worker)
ideally types like AudioBuffer would be Transferrable

Implementation ideas:

a WorkerAudioContext with the unsupported features removed (e.g. anything using media elements or media streams) but otherwise working identically to AudioContext
a WorkerAudioContext with all features, but some way of integrating with media elements/media streams from the main thread (e.g. by ID strings/object URLs)
a WorkerAudioContext with some kind of specialised WorkerMediaStream that can access an object URL created from the main thread and route audio from either a media element or media stream to the WorkerAudioContext
a WorkerAudioContext with all features, including speccing Media Streams for workers! (possibly very difficult)
a Transferrable AudioContextProxy that can be obtained from an existing AudioContext on the main thread and sent to a worker. Calls to AudioContextProxy would simply be forwarded to the existing AudioContext similar to what a postMessage implementation would do, so it's really using the same context. AudioContextProxy would also be missing media element/media stream features, but these could still be implemented from the main thread since it can use all features there. This is probably the simplest solution and significantly reduces the burden of bridging the APIs in JS (reducing the bridge to only the features necessary to cover media elements/media streams), but in the long term implementing MediaStreams in workers seems like it would be desirable too.