Web-audio-api: Allow downgrading of AudioParam automation rate

[JohnWeisz]

We are doing a lot of automation control in our Web Audio API based DAW. There can be entire user-configurable processing graphs that ultimately control one or more AudioParam objects at the end of the chain, something like this:

Live demo of this app

---

We simply call these things Nodes, and the purple color indicates an automation branch (in this screenshot, ultimately the gain AudioParam of a GainNode is being automated).

To maintain consistent performance and latency, we are using Web Audio API for our automation everywhere. For example, those values you see (0.25 and 0.5) are actually AudioBufferSourceNode instances outputting the given value constantly (as the ConstantSourceNode is not yet widely available), and the multiplier node (the one that says *) simply performs a multiplication using a GainNode (one signal is routed directly into the GainNode, while the other signal is routed into the gain AudioParam), and the result is finally connected to the gain AudioParam.

What we are doing here would be called sample-accurate automation, as this automation is applied to the final gain param at a-rate, which is essentially equal to the sample rate.

Now a lot of other, "native desktop" DAW suites offer disabling sample-accurate automation to squeeze out some more performance in heavy projects. I'm wondering if this is something the Web Audio API plans to offer, or already offers in some "hidden" way.

I'm aware there are _k-rate_ AudioParam objects, which are processed in sample _blocks_. This would be nice, but I've been checking out #49, and it seems you don't intend to add a k/a-rate switch (although that issue is 4 years old). This is understandable. However, what about processing AudioParam automation at a lowered sample rate? Is this something that would be feasible? We really don't need to automate a gain at 44.1 kHz, this is pure overkill for most tasks.

[padenot]

This is a good idea, but I can safely say we'll don't include this in the spec right now, since we've scoped the features we want for a Web Audio API v1 to make it into a candidate recommendation.

Implementations can (and do) take some steps to optimize some cases internally, but I agree that here, explicit is better than "magical".

[rtoy]

Here is a great example of why we might actually want to do this: https://jsfiddle.net/8vo0gbav/1/

Run this with Chrome 64 and Chrome canary. When dezipper = false (=> use setter), chrome uses about 30% CPU. Set dezipper = true (=> use setTargetAtTime), chrome uses > 100% CPU.

Chrome added support for a-rate biquad filter AudioParams quite some time ago, and removed dezippering from the value setter a month ago. (AFAIK, no one else has done a-rate biquads, but it is currently required in the spec.)

For people who need dezippering, there isn't a viable workaround other than replacing the filter with an AudioWorklet. And that's a terrible solution.

Adding an additional attribute to AudioParam to specify a-rate or k-rate is straightforward and the behavior for each value is well-understood. Transitions between the two would only take affect on render boundaries.

[mdjp]

We are going to consider moving this back to V.1 in the light of tests carried out by rtoy. More tests required and will discuss on the next call.

[rtoy]

Did a few more tests with Chrome in optimizing BiquadFilter
automations. The changes I've made improve some things and might
possibly get the CPU usage from 100% down to 70% at best, but this is
a long way from 35% CPU that the value setter code with dezippering
uses.

And the optimizations that I did only apply if the automation is
slowly changing from frame to frame (and also introduce probably
significant rooundoff errors at the same time).

At this point I think we have these options:

• Ignore this completely
• Put back dezippering. That's a huge change in the spec because it
  was never specified precisely, and Firefox doesn't have dezippering
  anyway.
• Add an attribute to AudioParam that allows the user to specify
  whether they want a-rate or k-rate. I think this is a relatively
  small change to the spec.

I'm leaning twoards the last option.

[mdjp]

Thank you for spending the time on this @rtoy . I agree that the last option seems the most sensible. I assume that this would allow optimisations for other nodes also if a-rate is not required, so a win win really.

[rtoy]

Yeah, it would apply to all AudioParams. We already have a table for each AudioParam that defines the rate. We'd just need to change that to say it's the default value for the attribute.

I think the only thing we need to say explicitly is that PannerNode using HRTF are always k-rate, independent of the setting. I think ABSN.playbackRate is currently spec'ed to be k-rate, but there's an issue saying that maybe we should make it a-rate. This change makes that possible and leaves it up to each browser to implement that.

[rtoy]

All of the AudioParams of a DynamicsCompressorNode are k-rate. It probably doesn't make sense to allow a-rate modifications of these, so maybe we would want to say it's illegal to set them to a-rate?

[karlt]

Linking https://github.com/WebAudio/web-audio-api/issues/509#issuecomment-94731355
as I assume that moderate-frequency parameter modulation was the motivation for a-rate Biquad params.

I don't know whether that is a common use case for Biquads, but I suspect
de-zippering would not be an ideal solution there.

For more typical biquad usage, I'd guess there's no need for sample-smooth changes in parameters?

[rtoy]

Yes, that was the motivation. We knew it would be expensive, but we failed to realize that with the removal of dezippering for biquads, the alternative setTargetAtTime would become hugely expensive.

[rtoy]

Proposal:

enum AutomationRate {
  "a-rate",
  "k-rate"
};

And add this attribute:

  AutomationRate automationRate;

The defaults are already given in the audio param tables. I think the only that needs to be specified is when you can't really change the rate:

• PannerNode with HRTF panning should ignore this setting; it's always k-rate. (This is already mentioned in the spec; just need to say it's ignored for HRTF. This also affects the AudioListener since that's intimately related to the panner.)
• DynamicsCompressorNode AudioParams are all k-rate today. I think we should throw errors if you try to change them to a-rate.
• AudioBufferSourceNode.detune and playbackRate are k-rate. Seems reasonable to allow a-rate if desired. This requires changing the playback algorithm somewhat.

I believe those are the only AudioParams that are not currently a-rate. Allowing existing AudioParams to be k-rate should be straight-forward.

[joeberkovitz]

Just wondering if it makes sense to also consider allowing BiquadFilterNode to take an option that controls the k-rate-ness of all of its AudioParams. Less general to be sure, but perhaps less likely to have unforeseen side effects.

[rtoy]

As mentioned in the call, @joeberkovitz suggestion implies that the constructor now has something that cannot be done with the existing factory method. Perhaps that's ok.

[rtoy]

For simplicity, I'm going to say ABSN AudioParam's cannot be changed from k-rate. Too much work in modifying the algorithm and if it really is desired, it can be done in V2.

[andrewmacp]

Just chiming in here as this came up in the context of a performance issue we hit in Chrome. We realized while updating our code to do manual dezippering that there is a big performance hit when automating a-rate Biquads, especially on low-end devices. We have a pretty big chunk of our 2.5m users on Chromebooks in an educational context and this effectively means that we can't do automation there, I can provide more concrete numbers or data if necessary.

It would be really valuable to be able to control the automation rate so that Web Audio developers can scale performance depending on available computing power. On a high-end machine we'd like to provide professional-quality audio, but we'd also like to provide a usable experience on weaker devices (Chromebooks, Windows 10 educational devices, mobile).

I definitely realize that it's late in the v1 spec process but the concern is that if this is left out of v1 then the performance of Biquads (and maybe other a-rate params) will be a blocker to supporting low-end devices, especially as other browsers catch up to the spec.

Let me know if I can provide any more info here and thanks a lot for bringing this up!

[rtoy]

From the teleconf today, there's general agreement to https://github.com/WebAudio/web-audio-api/issues/1269#issuecomment-370863696, basically throwing errors for things that are already defined to be k-rate.