Aframe: Preloading sound asset seems buggy on master

[pjoulot]

Hello,
I think the sound example in the documentation is broken on master. It works perfectly on the 0.5.0 release but not on the latest commit of the master branch.

• A-Frame Version: master
• Platform / Device: Windows 10 - Firefox Nightly
• Reproducible Code Snippet or URL: https://aframe.io/docs/0.5.0/components/sound.html#preloading-a-sound-asset

When I do the exact same thing from the documentation on master, I have this error in the console:

TypeError: Argument 1 of BaseAudioContext.decodeAudioData does not implement interface ArrayBuffer.

Thank you for reading this.

[ngokevin]

Thanks, currently looking at it. Having trouble getting Firefox to play sounds at all.

[ngokevin]

I had an issue with a certain sound where I had to re-encode it with ffmpeg just for Firefox. Does it work for you if you try other sounds?

[aas395]

This happens to me when I preload an audio file and then try to play the sound using .playSound(), but everything plays fine when the audio isn't preloaded and autoplay is set to true. I'm looking around in the source but I wanted to let you know in case that rang a bell for you. I know you guys reworked the way assets were cached between 0.5.0 and 0.6.0

[UXVirtual]

Same issue here. I get the following error when adding preload=auto to the <audio> element where I'm also using autoplay: true on the sound component that plays the preloaded sound in the scene:

Uncaught (in promise) TypeError: Failed to execute 'decodeAudioData' on 'BaseAudioContext': parameter 1 is not of type 'ArrayBuffer'.

This is with using ogg encoded audio files. It's also worth noting that manually calling sound.playSound() after the render loop begins produces the same error.

[johnshaughnessy]

Using THREE.Cache was added to resolve an issue where images were being downloaded more than once: https://github.com/aframevr/aframe/issues/2472, but unfortunately I think this code doesn't work for audio (and possibly video, but I'm not sure).
If you set preload=auto in an audio element, then aframe sticks the loaded audio _element_ into THREE.Cache once the browser finishes downloading the audio:
https://github.com/aframevr/aframe/blob/master/src/core/a-assets.js#L165
Later when the sound component uses the THREE.FileLoader to load the audio, it pulls the audio element from the cache and throws the error @HAZARDU5 linked above:
Uncaught (in promise) TypeError: Failed to execute 'decodeAudioData' on 'BaseAudioContext': parameter 1 is not of type 'ArrayBuffer'.
(The loader returns the cache hit here: https://github.com/mrdoob/three.js/blob/master/src/loaders/FileLoader.js#L42).
I think sticking the audio element into the THREE.Cache is a bug, because what we actually wanted in there was an ArrayBuffer, not an HTMLElement. A workaround is to use a-asset-item instead of an audio element inside of a-assets, and set the response-type to arraybuffer. That way, a-asset-item will load the audio using THREE's file loader and the Cache will have an ArrayBuffer:
https://github.com/aframevr/aframe/blob/master/src/core/a-assets.js#L109

e.g. <a-asset-item id="mySound" src="assets/audio/mySound.wav" response-type="arraybuffer"></a-asset-item>

@ngokevin I suggest the documentation for the sound component be changed to reflect this too, because as-is, the preloading audio example does not work:
https://aframe.io/docs/0.7.0/components/sound.html#preloading-a-sound-asset

[dmarcos]

@johnshaughnessy Should we store the file and not the element in the cache? I cannot remember why we store the element. I will have to dig into it. @ngokevin Do you remember the rational?

[johnshaughnessy]

@dmarcos I think we'd want to store an arraybuffer, but since the browser is the one loading the audio for an audio tag, I don't know of a way to get an arraybuffer out of the audio element. The a-asset-item workaround works because the THREE.FileLoader is makes an XMLHttpRequest, and that request is configured to return an arraybuffer: https://github.com/mrdoob/three.js/blob/master/src/loaders/FileLoader.js#L253

[madebynoam]

I also run into this bug (in 0.7.1) with preload="auto" and then trying to do audio.play() in a click handler (so that it activates the audio on mobile).

FWIW I'm able to use this workaround, based on @johnshaughnessy recommendation:

<a-asset-item id="mySound" src="assets/audio/mySound.mp3" response-type="arraybuffer"></a-asset-item>

Then instead of trying to do audio.play() in a click handler, I instead look up the element with the sound component and do:
soundEntity.components.sound.stopSound(); soundEntity.components.sound.playSound();

From some reason the stopSound call is necessary for it to work on iOS (~11.2).

[weddingdj]

@madebynoam Worked for ogg files in A-Frame 0.8.2 as well