Web-audio-api: OfflineAudioContext progressive rendering

[mudcube]

Has there been consideration to adding an onProgress event to the OfflineAudioContext rendering? It seems this would be very useful when mixing down larger songs to allow users to know how long the process is going to take.

[joeberkovitz]

Also consider this comment by Srikumar (originally in #69) that feels very significant to me:

Synchronous graph manipulation is, I'd say, a must have in the offline case. If I want to mix down a 5 minute composition with 8 tracks with 1 voice in each track triggered every second (not too taxing a scenario for the realtime composer case), that would need 2400 nodes to be pre-created ... and this doesn't count effects nodes. Permitting the graph to be changed within a script node would provide a way to instantiate these voices just in time.

If that isn't possible, it would help to add a separate synchronous event that can be used for this purpose, that would also be useful for tracking progress of an offline context.

[joeberkovitz]

A couple more points here:

1. Incremental construction for _online_ rendering will likely already have been implemented in setInterval/setTimeout handlers that build nodes needed for a window of AudioContext time in the near future. To conveniently do this for offline rendering, it would be great if the "progress" events here behaved similarly but with respect to _offline rendering time_ -- i.e., they fire when a given time interval in rendering time has elapsed, instead of an interval in clock time.
2. To allow incremental construction of audio nodes for offline rendering in a handler for such an event, without the offline context "racing ahead" in its own thread, it's necessary to pause the rendering while the handler is active.

[joeberkovitz]

TPAC resolution: Introduce a way to tell OfflineAudioContext to pause automatically at some predetermined interval in context rendering time units. The resulting state change event can be used by applications to notify when the context has paused. This also serves to notify of progress. Explicit resume is required to make the context start resuming again.

[cwilso]

Per #21, we should enable a way to get partial output too.

[joeberkovitz]

473 (now closed) proposed that another way of dealing with this issue is to have a single event callback on a context (either realtime or offline) that can be fired at some designated audio context time. It could either replenish a realtime graph that continues to run, or [synchronously] replenish an offline graph that is paused during the event dispatch.

[hoch]

For the moment, I am exploring with few ideas on pausing OfflineAudioContext, which turned out to be super-useful for the API testing.

(1) onpause event: although it is an old pattern, but it is great for the recurring routine with an interval.

var context = new OfflineAudioContext(2, 441000, 44100);

context.pauseAtTime(2.0);

context.onpause = function (event) {
  // change your graph or do something with the partial buffer.
  var channelData0 = event.renderedBuffer.getChannelData(0);
  // ...and schedule the next pause.
  context.pauseAtTime(event.playbackTime + 2.0);
}

context.startRendering();

(2) pauseAtTime() promise pattern: this pattern is useful when there are specific tasks for multiple time points.

var context = new OfflineAudioContext(2, 441000, 44100);

context.pauseAtTime(2.0).then(function (buffer, playbackTime) {
  // do stuffs here (1).
});

context.pauseAtTime(6.5).then(function (buffer, playbackTime) {
  // do stuffs here (2).
});

context.startRendering();

In my opinion, we need both patterns - but probably (hopefully) we can have something much better pattern than these.

[joeberkovitz]

I agree that both patterns are needed. Aside from ease of use, the first pattern is necessary to avoid infinite closure nesting from calling pauseAtTime() inside a pauseAtTime completion handler.

[hoch]

@joeberkovitz Yes. That's was the primary reason for the pattern #1.

[hoch]

Now I am thinking this is just a variant of ScriptProcessorNode behavior. Glad that I found that it is already implemented, but we'll have to move it so only OfflineAudioContext can use it with onpause or pauseAtTime().

[hoch]

By the way, the browser implementors will greatly benefit from this. OfflineAudioContext is the only way to test the implementation with sample precision and this feature is necessary to build a more reliable testing framework for the API.

For example, we can't change the audio graph in the middle of rendering in a sane way. So this is one of workaround:

context.onstatechange = function () { // Finger crossed! };
context.startRendering().then(verifyResult);

The problem is we don't know when the statechange event will be fired. In the worst scenario, the event might not be fired until the rendering is finished and the test becomes flaky.

I will write up the proposal for this at some point, but I would love to hear what WG think.

[joeberkovitz]

@hoch I didn't understand your comment about it being already part of ScriptProcessorNode behavior. What is the piece of this that is already implemented? I don't see it in any of the specs; are you referring to some private implementation of pause behavior in Blink?

In any case I think the pause behavior is essential for progressive rendering of offline contexts in general, and would also like to see WG comments.

[hoch]

@joeberkovitz What I meant was this is actually a same pattern with OfflineAudioContext with a special ScriptProcessorNode internally hooked up. The only difference is the onaudioprocess event will be triggered at a user-specified time from pauseAtTime() call. I guess this is just one of implementation detail - not that important from the spec's perspective.

Perhaps it should look like this?

interface OfflineAudioContext : AudioContext {
  Promise<AudioBuffer> startRendering();
  Promise<AudioBuffer> pauseAtTime(double time);
  attribute EventHandler oncomplete;
  attribute EventHandler onpause;
};

interface OfflineAudioPauseEvent : Event {
  readonly attribute AudioBuffer renderedBuffer;
  readonly attribute double playbackTime;
};

interface OfflineAudioCompletionEvent : Event {
  readonly attribute AudioBuffer renderedBuffer;
};

OfflineAudioPauseEvent should contain the actual (sample accurate) playbackTime. I am not sure how I can incorporate this into Promise pattern.

My only concern is the timing of onpause event being fired won't be sample accurate because we can't trigger this in the middle of the rendering quantum. It has to be around the block boundary - but that's something I can live with.

[joeberkovitz]

@hoch I don't think there is really any requirement that pauseAtTime() be able to create an artificial block boundary at any desired sample frame.

I propose that we specify that a pause always occurs at the first block boundary at or following the requested context time, or something like that. The time reported in the event should be sample accurate, but it will always be the time of the first sample frame in the next block that will be rendered (i.e. the sample frame right after the last one that was processed).

[hoch]

The time reported in the event should be sample accurate, but it will always be the time of the first sample frame in the next block that will be rendered

@joeberkovitz Yes, that is exactly what I am thinking.

[joeberkovitz]

Seems to be ready for editing...

[hoch]

Actually, I am exploring the alternate way of doing this: using suspend(time) and resume() in OfflineAudioContext. Maybe I can come up with two proposals and we can start discuss from there.

[rtoy]

I prefer this approach because it makes it very similar to an online
context.

On Thu, Apr 30, 2015 at 8:38 AM, Hongchan Choi [email protected]
wrote:

Actually, I am exploring the alternate way of doing this: using
suspend(time) and resume() in OfflineAudioContext. Maybe I can come up
with two proposals and we can start discuss from there.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/302#issuecomment-97842928
.

Ray

[hoch]

Here's what I have been experimenting with:

Pattern 1: using onstatechange event handler

var oac = new OffilneAudioContext(1, 44100, 44100);

oac.onstatechange = function (event) {
  if (oac.state === 'suspended') {
    var data = event.renderedBuffer.getChannelData(0);
    var now = event.playbackTime;
    // do things here.
  }
  oac.resume();
  oac.suspend(event.playbackTime + 0.2);
};

oac.startRendering().then(handleCompletion);
oac.suspend(0.2);

Pattern 2: using promise from suspend().

var oac = new OffilneAudioContext(1, 44100, 44100);

oac.startRendering().then(handleCompletion);

oac.suspend(0.2).then(function (renderedBuffer, playbackTime) {
  var data = renderedBuffer.getChannelData(0);
  var now = playbackTime;
  // do things here.
  oac.resume();
});

It complicates the current audio context little bits because its suspend() method should support an argument on both online and offline modes.

One thing that bothers me with this approach is when user forgets to call resume() method. In such case, OfflineAudioContext will never end and the thread will be hang forever. So while waiting for resume we can make OAC yields processing time for other higher priority work (this is okay because we already cannot specify the resume timing in a sample accurate way) and make it stop within a reasonable amount of time. Anyhow, this is just a technical detail.

I believe this needs more discussion: especially about which is more reasonable and practical. I can see the practicality of recycling suspend/resume, but also I like the simplicity of pauseAtTime and not allowing it on the realtime context.

[rtoy]

A few comments.

In pattern 1, I don't think events have a playbackTime. Are you suggesting that be added? Can't it be obtained from the event.srcTarget (or other attributes)? Also, I think you want to call suspend before resuming. If resume is fast, and GC happens before suspend, you might miss the the suspend time.

Likewise, I think you need to call suspend before startRendering.

Finally, suspend(t) on a online context needs further work. Perhaps we can just disallow suspend(t) for an online context?

[hoch]

@rtoy Yes, I didn't clarify that. I think we need another type of event like AudioProcessingEvent from ScriptProcessorNode.

Though I am still not sure how useful passing a partial rendered buffer is. Perhaps you can investigate the buffer content and do audio graph manipulation or even change the buffer content. At this point, I am just handwaving.

[joeberkovitz]

I think generating a partially rendered buffer is important for offline contexts. Not so much for examining it, but more for sending it to some non-real-time destination (say, a file, or a WebSocket, or a Blob). Without partial rendering, one can't have a scalable approach to offline graph rendering over very long time frames.

Furthermore a partially rendered buffer should not start at t=0, or else these buffers will just become bigger and bigger over a very long time frame, the longer rendering progresses. Instead, partially rendered results should cover only a time interval between the last partial-render and the current one, so that rendering results can be broken up into manageable chunks.

[hoch]

I think generating a partially rendered buffer is important for offline contexts. Not so much for examining it, but more for sending it to some non-real-time destination (say, a file, or a WebSocket, or a Blob). Without partial rendering, one can't have a scalable approach to offline graph rendering over very long time frames.

@joeberkovitz This can be a huge benefit if we can have Web Audio API implementation on Node.js. A server with Node.js can be effectively an audio rendering farm this way.

[hoch]

In my current experimental build, the following code suspends the offline audio context every 0.1 second. (Technically it is about 34 rendering quanta in 44100Hz)

var oac = new OffilneAudioContext(1, 44100, 44100);

oac.onstatechange = function (event) {
  if (oac.state === 'suspended') {
    console.log('suspended at = ' + context.currentTime);
    oac.suspend(oac.currentTime + 0.1);  
    oac.resume();
  }
};

oac.suspend(0.1);
oac.startRendering();

And this is the result from the console:

suspended at = 0.09868480725623582
suspended at = 0.19736961451247165
suspended at = 0.2960544217687075
suspended at = 0.3947392290249433
suspended at = 0.49342403628117915
suspended at = 0.592108843537415
suspended at = 0.6907936507936508
suspended at = 0.7894784580498866
suspended at = 0.8881632653061224
suspended at = 0.9868480725623583
suspended at = 1.0855328798185941
suspended at = 1.18421768707483
suspended at = 1.2829024943310658
suspended at = 1.3815873015873017
suspended at = 1.4802721088435373
suspended at = 1.5789569160997732
suspended at = 1.677641723356009
suspended at = 1.7763265306122449
suspended at = 1.8750113378684807
suspended at = 1.9736961451247166

As @rtoy's comment, suspend() should be called before resume() or startRendering(). This is because OAC runs very fast, so scheduling a suspend might happen after the specified time passed. We can discuss on the right behavior for this case, but I am assuming scheduling a suspend in the past should not be allowed.

[hoch]

@joeberkovitz After the discussion with @rtoy and @cwilso, we believe introducing 'partially rendered buffer' in V1 would be difficult. Probably it will cause major changes in OfflineAudioContext's constructor and the behavior - and we don't want that in V1.

In short, I would like to keep this feature about manipulating audio graph during the rendering with the render-block-size precision. At this point, opinions from WG would be really helpful to push this feature out there since there are some details that need to be flattened out.

[hoch]

Here's the short version of my proposal. Eventually I will post a PR as a spec change, but hopefully we can start discussion from this.

The IDL change

[Constructor(unsigned long numberOfChannels, unsigned long length, float sampleRate)]
interface OfflineAudioContext : AudioContext {
    Promise<AudioBuffer> startRendering();
    Promise<void> suspend(double when);
    Promise<void> resume();
                attribute EventHandler oncomplete;
};

Notes

1. suspend() has _non-optional_ 1 argument: it will be quantized by the size of render quantum.
2. Scheduling a suspend in the past or beyond the total render duration is NOT allowed.
3. Scheduling multiple suspends at the same render block onset is NOT allowed.
4. Resuming a context that is not suspended or started is NOT allowed.
5. MUST reject the promise when suspend/resume() fails by violating 1, 2, 3 or 4.

Patterns

(1) Promise-recurring pattern

var context = new OfflineAudioContext(2, 441000, 44100);

function onsuspended() {
  context.suspend(context.currentTime + 0.1).then(onsuspended);
  context.resume();
}

context.suspend(0.1).then(onsuspended);
context.startRendering();

(2) Promise-sequential pattern

var context = new OfflineAudioContext(2, 441000, 44100);

context.suspend(0.2).then(function () {
  context.resume();
});

context.suspend(1.2).then(function () {
  context.resume();
});

context.startRendering();

(3) Event handler pattern

var context = new OfflineAudioContext(2, 441000, 44100);

context.onstatechange = function () {
  if (context.state === ‘suspended’) {
    context.suspend(context.currentTime + 0.1);
    context.resume();
  }
}

context.suspend(0.1);
context.startRendering();

[rtoy]

Just one additional note. To be predictable, you should call suspend()
once before calling startRendering(). It's not illegal for the first
suspend to be called after startRendering, but since offline contexts can
be very fast, it's pretty hard to guarantee suspend(t) will actually
suspend at time t since it may have already passed.

On Wed, May 27, 2015 at 3:47 PM, Hongchan Choi [email protected]
wrote:

Here's the short version of my proposal. Eventually I will post a PR as a
spec change, but hopefully we can start discussion from this.

_The IDL change_

[Constructor(unsigned long numberOfChannels, unsigned long length, float sampleRate)]
interface OfflineAudioContext : AudioContext {
Promise startRendering();
Promise suspend(double when);
Promise resume();
attribute EventHandler oncomplete;
};

_Notes_

1. suspend() has _non-optional_ 1 argument: it will be quantized by the
   size of render quantum.
2. Scheduling a suspend in the past or beyond the total render duration is
   NOT allowed.
3. Scheduling multiple suspends at the same render block onset is NOT
   allowed.
4. Resuming a context that is not suspended or started is NOT allowed.
5. MUST reject the promise when suspend/resume() fails by violating 1, 2,
   3 or 4.

_Patterns_

(1) Promise-recurring pattern

var context = new OfflineAudioContext(2, 441000, 44100);

function onsuspended() {
context.suspend(context.currentTime + 0.1).then(onsuspended);
context.resume();
}

context.suspend(0.1).then(onsuspended);
context.startRendering();

(2) Promise-sequential pattern

var context = new OfflineAudioContext(2, 441000, 44100);

context.suspend(0.2).then(function () {
context.resume();
});

context.suspend(1.2).then(function () {
context.resume();
});

context.startRendering();

(3) Event handler pattern

var context = new OfflineAudioContext(2, 441000, 44100);

context.onstatechange = function () {
if (context.state === ‘suspended’) {
context.suspend(context.currentTime + 0.1);
context.resume();
}
}

context.suspend(0.1);
context.startRendering();

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/302#issuecomment-106101885
.

Ray

[karlt]

Hongchan Choi writes:

@joeberkovitz After the discussion with @rtoy and @cwilso, we believe
introducing 'partially rendered buffer' in V1 would be difficult. Probably it
will cause major changes in OfflineAudioContext's constructor and the behavior

• and we don't want that in V1.

Would it really be better to make these major changes in V2, after
the OfflineAudioContext has been further modified?

Having to produce a complete recording of audio in the
OfflineAudioCompletionEvent seriously reduces the useful lifetime
of a context, so a solution will be required at some point.

When partial rendering is supported, having to know a priori the
total rendering length would also be a limitation.

In short, I would like to keep this feature about manipulating audio graph
during the rendering with the render-block-size precision. At this point,
opinions from WG would be really helpful to push this feature out there since
there are some details that need to be flattened out.

Are you sure that adding on a graph manipulation feature without
considering partial and variable length rendering at the same time
will not make adding this later even more awkward?

One alternative to consider instead of suspend/resume may be to
allow restarting an OfflineAudioContext that has queued
OfflineAudioCompletionEvent and resolved the startRendering()
Promise.

If a fixed rendering quantum is acceptable, then this requires
little change in API. If variable length quanta are required then
some new API is necessary.

[hoch]

@karlt First off, I think this is not about the 'variable length' or 'ranged' rendering. The partial rendered buffer that @joeberkovitz mentioned was getting the rendered content from the previous resume point and the last suspend point. But your idea of the variable length rendering is also interesting too.

Are you sure that adding on a graph manipulation feature without
considering partial and variable length rendering at the same time
will not make adding this later even more awkward?

Why is it awkward? The graph manipulation by the block boundary is what is happening with the real-time audio context, and the offline one also should work in the same fashion so you can get the same result on both modes with the same code.

If a fixed rendering quantum is acceptable, then this requires
little change in API. If variable length quanta are required then
some new API is necessary.

Isn't this what my proposal implies? These little changes are meaningful for few strong use cases. Also the partial rendering idea is more or less being able to stream rendered data, so I believe we need a completely different interface rather than OfflineAudioContext.

I am not strongly opposed to the idea of partially rendered buffer, but I would like to keep the render quantum fixed for both audio contexts.

[joeberkovitz]

It seems OK to me to maintain a consistent render quantum for both AC types. It seems less good to have to predict the total length up front. It doesn't seem good at all to be forced to buffer the entire rendered output audio result in memory before it can be written anywhere.

I guess I don't see yet why we need a completely different interface from OfflineAudioContext in order to be able to receive the rendered data in chunks. Let's discuss the question at the F2F.

[joeberkovitz]

F2F comment: we discussed that it's not desirable to make the mechanism for suspend/resume (which centers around incremental graph construction) also serve to break up rendered runs of samples for streaming. Also streaming would ideally be directed into a codec of some kind, or MediaRecorder. These can be separate v.next additions.

[hoch]

The pull request on this issue is here: 4152b1a9f745f879e22ccfc1b7c7a7c38b9f6ff6

[rtoy]

Suspend/resume for offline contexts addressed in #561.

[hoch]

Closing this by https://github.com/WebAudio/web-audio-api/pull/561.

[JohnWeisz]

I'm not quite sure what the problem is here. AFAIK, you can simply read audioCtx.currentTime in a requestAnimationFrame loop to query the progress of the rendering. Divide that with the total duration of your OfflineAudioContext, and you have the value as a percentage in the [0, 1] range.

No need to repeatedly call suspend and resume for this task.