Web-audio-api: Expose whether AudioBufferSourceNode.start() succeeded

[mounirlamouri]

/CC @rtoy @padenot @jernoble

As far as I can tell, AudioBufferSourceNode.start() is a no-op on Safari iOS unless the call came from a user gesture. Chrome is considering to do the same in order to have more consistency with regards to autoplay behaviour on mobile. However, the start() method doesn't expose if the call was successful which makes me uncomfortable doing this change.

Would it make sense to have a pattern similar to HTMLMediaElement.play() for AudioBufferSourceNode and return a Promise that rejects if the playback isn't allowed and fulfil when the playback starts?

[mounirlamouri]

By AudioBufferSourceNode, I actually meant AudioBufferSourceNode and OscillatorNode, I guess :)

[padenot]

Allocating a Promise for each AudioBufferSourceNode might be a bit expensive. It is not unusual for an application to create thousands of AudioBufferSourceNode in bursts, and creating an AudioBufferSourceNode and scheduling its start must remain very very cheap.

How about we make {AudioBufferSourceNode,OscillatorNode}.start() return a bool that indicates whether the playback is going to effectively start or not, or maybe some other lightweight mechanism ?

[mounirlamouri]

I have no strong opinion between a bool and a Promise. I find the promise model cleaner but the bool might be simpler and more efficient. I guess if there is no need to fulfil the promise when the playback actually starts, the bool might be feature-wise the same.

[rtoy]

I think it would be nicer to have a method on an AudioContext that returns a promise that is resolved when audio can actually be heard. Having to check the result of start() to see if it would play seems painful, and too late---you wanted to start something and found out that you can't.

[mark-buer]

Expose whether AudioBufferSourceNode.start() succeeded

It already is.

It is possible to detect if the audio system is "primed" on mobile Safari/WebKit.

The AudioContext is initially suspended. It remains in that state until such time as audio playback is initiated from within a 'touchxxx' event (perhaps other events too - haven't checked the code), whereupon it transitions to the running state.

Thus, code can examine the audioContext.state property to determine if starting playback was successful.
Additionally, code can be notified of this transition by observing the promise returned by audioContext.resume().

I'm pleased this topic, touch activation of web audio, is being discussed by the specification team. Standardisation of the behaviour, in whatever shape, is highly desirable.

Here is some demonstrative code targeted at mobile Safari:

var audioContext = new webkitAudioContext();

audioContext.resume().then(function() {
    console.log('resume promise is resolved');
});

document.body.addEventListener('touchend', activate);
function activate() {
    console.log('touchend event did fire');
    startOscillator();
    if (audioContext.state === 'running') {
        document.body.removeEventListener('touchend', activate);
    }
}

function startOscillator() {
    console.log('state 1: ' + audioContext.state);
    var oscillator = audioContext.createOscillator();
    console.log('state 2: ' + audioContext.state);
    oscillator.frequency.value = 420;
    console.log('state 3: ' + audioContext.state);
    oscillator.connect(audioContext.destination);
    console.log('state 4: ' + audioContext.state);
    oscillator.start(0);
    console.log('state 5: ' + audioContext.state);
    oscillator.stop(audioContext.currentTime + 1);
    console.log('state 6: ' + audioContext.state);
}

// audioContext.createOscillator();

Produces the output:

touchend event did fire
state 1: suspended
state 2: running
state 3: running
state 4: running
state 5: running
state 6: running
resume promise is resolved

However, the behaviour differs depending on whether any prior attempt was made to create an audio node outside of the touch event handler.
Uncommenting the final line of code in the above sample produces this output:

touchend event did fire
state 1: suspended
state 2: suspended
state 3: suspended
state 4: suspended
state 5: running
state 6: running
resume promise is resolved

[mounirlamouri]

@rtoy the issue with your proposal is that it doesn't expose to the user that the attempt to play failed because if start() was ignored, that promise will never actually fire, right? Also, isn't state set to running actually exposing that change? (honest question, I don't know the Web Audio spec that well)

@mark-buer you are correct: Safari iOS provides a way to check but it's very hacky and sounds like a violation of the spec because it sets the state as running earlier than it should if there is a user gesture requirement. In other words, I think that this behaviour will be very different between Safari on iOS and on MacOS.

[AshleyScirra]

Please don't copy Safari's behavior, it's annoying! Why not just let the full Web Audio API work but start in a muted state, and automatically unmute it in the first user gesture? That is exactly what any libraries/frameworks/people who really do want to play audio will do. The whole "audio fails until first gesture" model is a pain in the ass for development because you have to buffer all possible audio actions that the user can start until the first user gesture when you then really start them. This is also a portability problem from desktop to mobile. If it just started muted then automatically unmuted, there are zero code changes required, you still avoid annoying the user, and nobody can do anything they couldn't before.

So I think the spec should just say "the user agent may mute the output at its discretion" and don't bother with any of this laborious "suspended" or "playback not allowed" state which just pointlessly creates more work for developers.

[AshleyScirra]

I also have to add that iOS Safari recently changed the unblocking mechanism to only work in touchend instead of touchstart, which broke all existing web content which used our original unblocking workaround. Just unmuting at the user agent's discretion avoids this kind of breakage when changing precisely when web content is allowed to unblock audio.

[padenot]

TPAC resolution:

• An AudioContext starts in "suspended" state
• The UA can delay the first "suspended" => "running" transition arbitrarily
• Authors can programmatically know if the AudioContext has been granted the audio output access by inspecting the state property of the AudioContext.
• Authors can provoke the transition to "running" by calling resume() on the AudioContext, in conditions that are at the discretion of the UA. For example, from a touch handler.

@mounirlamouri, does that accurately capture what we talked about ?

[mounirlamouri]

Yes, I'm going to write a PR that changes the spec accordingly.