Three.js: AudioLoader doesn't behave correctly without user interaction

[funwithtriangles]

The AudioLoader makes use of window.AudioContext. This is problematic because any use of audio in Chrome requires a user interaction. While that's normally solvable by triggering audio with a click, this isn't suitable when dealing with asset loading, as this sort of thing should start the moment the page loads.

For example, I'd like to make use of AudioLoader when the page first loads, to download audio files along with other assets. If I do this, Chrome may bring up this warning (although this seems to be intermittent):

The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu

To be clear, I have a click event for playing the sounds. Currently, we need a click event even when _loading_ the sounds.

Looking at the source code for AudioLoader, this is happening because it's trying to load and decode the audio in the same step.

There doesn't seem to be a clear solution on how to fix this. It's a real pain that Chrome is restricting even the decoding of audio. Perhaps audio files need to be treated as normal files while downloading, and then there is a separate step (after a user interaction to start the app) for the audio to be decoded.

Tested on Chrome, Windows. Three r116.

[funwithtriangles]

Here's a rough workaround I've put together, TwoStepAudioLoader. The load method loads the file using FileLoader and also stores the data for use later. The decodeAll method can be called at a later time, and takes all previously loaded files and decodes them.

import { FileLoader, Loader, AudioContext } from 'three'

function TwoStepAudioLoader (manager) {
  Loader.call(this, manager)
  this.rawBuffers = []
}

TwoStepAudioLoader.prototype = Object.assign(Object.create(Loader.prototype), {

  constructor: TwoStepAudioLoader,

  load: function (url, onLoad, onProgress, onError) {
    let loader = new FileLoader(this.manager)
    loader.setResponseType('arraybuffer')
    loader.setPath(this.path)
    // Load files and store the raw uncoded data in an array, with url information
    loader.load(url, buffer => {
      let bufferCopy = buffer.slice(0)

      this.rawBuffers.push({
        data: bufferCopy,
        key: url,
      })
      onLoad(bufferCopy)
    }, onProgress, onError)
  },
  decodeAll: function (onLoad) {
    let context = AudioContext.getContext()
    let promises = []

    this.rawBuffers.forEach(function ({ data }) {
      promises.push(context.decodeAudioData(data))
    })

    // Decode all the buffers and store as an object with URLs as keys
    Promise.all(promises).then(audioBuffers => {
      const allBuffers = {}
      audioBuffers.forEach((buffer, index) => {
        const key = this.rawBuffers[index].key
        allBuffers[key] = buffer
      })
      onLoad(allBuffers)
    })
  },
})

export { TwoStepAudioLoader }

Usage:

const audioUrl = 'someAudio.ogg'
const loader = new TwoStepAudioLoader()

loader.load(audioUrl, () => {
   // All files are now loaded, so app can start on click
   document.body.addEventListener('click', () => {
      // Now decode the audio
      loader.decodeAll(allBuffers => {
         const myBuffer = allBuffers[audioUrl]
         // Use the buffer as you like
      })
   })
})

[funwithtriangles]

Inviting @Mugen87 and @meatwallace (recent contributors) on their thoughts. :)

[Mugen87]

There doesn't seem to be a clear solution on how to fix this.

Well, the solution on app level is to provide e.g. a splash screen right before you start loading assets which requests a user interaction:

https://threejs.org/examples/webaudio_timing

It's a real pain that Chrome is restricting even the decoding of audio.

I suggest you discuss this policy directly with the browser vendor. I do not vote to make changes to AudioLoader.

https://bugs.chromium.org/p/chromium/issues/list

[funwithtriangles]

My concern is that other users would make the same mistake I made in thinking that AudioLoader can be treated like all the other loaders and can be used together with other loaders.

For instance, it seems quite natural to use something like a Promise.all() with many different loaders to manage the loading of all your assets, but this wouldn't work with AudioLoader. Perhaps the naming should be revisited or at least there could be some other solutions proposed in the documentation?

[Mugen87]

there could be some other solutions proposed in the documentation?

Enhancing the page of AudioLoader with a hint about this issue is a good idea (so referring to Chrome's audio policy and highlight the need for a user interaction before using the class).

[funwithtriangles]

I'm happy to do this. Do you think it would also help to add something like TwoStepAudioLoader to the examples to show how a user can download sounds on page load? I don't like the idea of telling user's they can't do something without offering some solution.

[Mugen87]

I don't like the idea of telling user's they can't do something without offering some solution.

IMO, the solution the library should promote is a splash screen. I see no need for an additional loader class.

[funwithtriangles]

So let's say you have a game with lots of sound effects along with other assets. With your proposed solution, the user has to wait for the normal assets to download, then click "start" before the rest of the audio assets download. Don't you think that's a bit of a poor user experience?

[Mugen87]

Well, you show the screen right from the beginning. It make sense in most cases anyway since it's recommended to tell the user stuff like: _Turn on audio speakers for best experience_.

[funwithtriangles]

Sure, I think a splash screen makes sense, but while it's displaying is usually the best time to load all of your assets. If the user clicks "start" and is then faced with another loading screen, this isn't ideal.

I'd agree that this is a matter of application design but a library can help with common patterns. Loading a second round of assets after clicking "start" is not a common pattern.

[funwithtriangles]

However I will agree that an extra loader seems like overkill. Just wondering if there are other options.

[Mugen87]

Maybe you can convince the Chromium project to alter its audio policy 😉

[Mugen87]

Anyway, feel free to improve the documentation as suggested 👍

[Mugen87]

Loading a second round of assets after clicking "start" is not a common pattern.

BTW: I never suggested such a thing. The idea was to start the entire loading process after a user interaction, not just the audio files. So the workflow is "Splash" or "Welcome" screen -> Interaction -> "Loading Screen" -> "Application Start".

[mrdoob]

@funwithtriangles You're saying that context.decodeAudioData requires user action? If so, can you create a jsfiddle? Also, does this reproduce in other browsers?