Web-audio-api: DecodeAudioData doesn't work on partial content

[nums]

Hello,

When we want to use it for audio streaming like media source api : https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html
The DecodeAudioData does not work when trying to decode a partial content while the audio tag play very well this content.

example : http://dashif.org/reference/players/javascript/0.2.0/index.html

[rtoy]

I believe that is the intent for decodeAudioData. If you want to stream data to WebAudio, you can use a MediaElementAudioSourceNode.

[nums]

Hello,

Thanks for your reply.

Through a MediaElementSource, I have to go through a JavaScriptNode to recover audiobuffer in real time. I did not find an other way to do that without JavascriptNode.
It will be great if Web Audio API could natively decode partial content.

[jernoble]

On Jun 19, 2014, at 2:41 AM, Emmanuel Freard [email protected] wrote:

Hello,

Thank you for your reply.

Through a MediaElementSource, I have to go through a JavaScriptNode to recover audiobuffer in real time. I did not find an other way to do that without JavascriptNode.
It will be great if Web Audio API could natively decode partial content.

I agree. It would be especially useful when paired with a queueing AudioBufferSourceNode, where partial output from the decoder could be appended to an existing node.

-Jer

[ghost]

I'm a bit late to this one but I can't see how the decoding of partial content could be done automatically with decodeAudioData(). File formats such as OGG Vorbis (and MP3 if I remember correctly) store the information required to decode the audio in the file header. The decoder could store that information somewhere but it would have no idea if N number of calls to decodeAudioData() were sequential and/or partial chunks from the same source file. Also, the structure of the source file needs to be considered, some formats can only be decoded in chunks (frames) of specific sizes.

It's actually possible to "stream" audio now using decodeAudioData() if the programmer understands the format of a particular audio file. I managed to do this with OGG Vorbis (load and decode it in chunks) a while ago by storing the required decoding information and prepending it to the beginning of each chunk prior to decoding. The data loading and splicing can be done in a worker and then transferred (zero-op) to the main thread for decoding.

That being said, the existing media elements do all of this already so you would need a really good reason to not use them for streaming, IHMO :)

[cwilso]

I _think_ we made the decision to push more powerful decoding API to v.next?

[padenot]

That's what I remember as well.

[rtoy]

Based on the comments in https://github.com/WebAudio/web-audio-api/issues/337#issuecomment-60962915 and https://github.com/WebAudio/web-audio-api/issues/337#issuecomment-60963610,

I think we've decided to move this to v.next. Adding Needs WG review for a final decision.

[samelie]

+1

Posted a question here

[jdsmith3000]

Discussed at today's Working Group call and agreed to move to v.next, along with issue #30.

[bjornm]

At Soundtrap, we have custom codecs using web assembly compiled ogg/vorbis. These are exposed in async streaming js apis to allow you to decode audio data in chunks. You specify the offset and number of frames and get an audio buffer back. This, in in turn can be used to skip or loop. It can run in its own js worker.

This streamed data is currently scheduled in 1s chunks in back-to-back AudioBufferSourceNodes to emulate an audio queue with gapless playback. While it works (and is fun!), I'm really looking forward to replacing that with a streaming decode audio buffer api and audio worklets.

If anyone in the working group is interested, I'd be happy to demo the whole use case and the code to better illustrate the problem we're trying to solve and the api's we ended up writing.

[bjornm]

Someone asked me to explain a bit more about the Soundtrap javascript audio codec library we use internally.

The audio codec library is written in dart / javascript / web assembly. It does not handle any networking aspect of the decoding / encoding, but it works on typed arrays (e.g. Float32Array and UInt8Array). These could come from the network, from memory or from indexed db, so they work in a variety of use cases. There are some abstractions in the api to allow for flexibility in implementation, pluggability and the language used.

AudioCodec
The AudioCodec class contains a registry of the available codecs for the platform and provides methods to get such decoders.
AudioCodec.canDecode('mp3') => true
AudioCodec.canEncode('flac') => false
etc

It acts as a pluggable registry so we can implement and register codecs in AudioCodec independently of any client code asking to use the codecs.
AudioCodec.registerEncoder(...class implementing AudioEncoder...)
AudioCodec.registerDecoder(...class implementing AudioDecoder...)

AudioDecoder and StreamingAudioDecoder
When wanting to decode some byte data:
var decoder = AudioCodec.getDecoder('mp3', bytes);
var metaData = decoder.getMetaData();
// metaData.channels
// metaData.sampleRate
// metaData.length
var offset = 0;
var length = 10000;
var chunk = await decoder.decodeChunk(offset, length); // returns a Float32Array[] with one array per channel of audio. Asynchronous promise returned
...
decoder.end();

Implementations
Streamable implementations of AudioDecoder provide the methods above - getMetaData() and decodeChunk(). Some of them are implemented in javascript (e.g. aif decoder), some in wasm (e.g. ogg/vorbis), some hybrid of javascript and web audio (using AudioContext.decodeAudio). In cases where we cannot (yet) stream audio, there is a simple AudioDecoder interface without the chunked methods, simply returning the full result directly.
var full = await AudioDecoder.decodeFully(...); // returns Float32Array[] or AudioBuffer

AudioEncoder and StreamingAudioEncoder
Similar interface to AudioDecoder but takes Float32Array and returns Uint8List. Depending on the codec, the constructor will take options specific to the codec (e.g. bitrate).
var encoder = AudioCodec.getEncoder(numChannels, sampleRate, options);
var chunk = await encoder.encodeChunk(channelData); // Passes in Float32Array[] and returns Uint8Array or ByteBuffer

As can be seen from the example above, the returned decoder is searchable, since you can specify offset and length for the decodeChunk(). This allows us to loop audio while playing back in a gapless fashion.

A limitation of the example above is that all encoded data is passed in the constructor, which prevents streaming from the network, and leads to waste of memory. A future version will replace the "bytes" argument with something like the Streams API, agnostic of memory / network / disk, where the encoded data stream can be searched and read in order to support the decodeChunk() call. e.g.
Stream {
Promise seek(int offset);
Promise read(int count);
Promise close();
}

[AnthumChris]

@nums I'm writing a demo and providing code examples for decoding audio in chunks with Streams. This started as a proof of concept to bypass the current limitation of decodeAudioData() requiring complete files: AnthumChris/fetch-stream-audio

@bjornm I took your advice and started with WAV decoding since it's easier and this is my first experience coding with audio. Good foresight.

[AnthumChris]

Here's a JavaScript Ogg Opus decoder that uses Wasm and libopusfile to decode streams in chunks:

https://github.com/AnthumChris/opus-stream-decoder

[mdjp]

This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662

[guest271314]

Cannot decodeAudioData() be modified to do this https://github.com/WebAudio/web-audio-api/issues/337#issuecomment-60258749

I managed to do this with OGG Vorbis (load and decode it in chunks) a while ago by storing the required decoding information and prepending it to the beginning of each chunk prior to decoding.

[hillct]

Cannot decodeAudioData() be modified to do this [#337 (comment)]

Almost certainly. The argument for implementing it directly within decodeAudioData() would be the significant performance benefit you'd get, relative to implementing stream filtering based upon certain file stream tags, with repeated calls to decodeAudioData() to achieve the same effect.

I managed to do this with OGG Vorbis (load and decode it in chunks) a while ago by storing the required decoding information and prepending it to the beginning of each chunk prior to decoding.

We used the same approach to achieve streaming (partial download) segmented decoding of MP3 files, but network performance variability rendered any buffering that can be implemented at the JS level, insufficient, to achieve reliable audio playback without drops and other artifacts.

As such, it need to be implemented at a lower level, within the API, where more reliable buffering can be achieved.

[guest271314]

@hillct

Consider https://github.com/xiph/opus-tools/issues/49 where MediaSource does not support audio/ogg;codecs=opus codec.

Am able to implement the pattern to an appreciable degree now https://github.com/WebAudio/web-audio-api/issues/2135 two different approaches. Using AudioWorklet by storing value of ReadableStream within an Array then re-reading the Array as a Unit8Array with correct offsets. Alternatively using MediaStreamAudioDestinationNode. Each approach has issues.

Attempted to use OfflineAudioContext though the implementation complains about not being able to use the data in a different context.

[guest271314]

@hillct FWIW, current code

  <audio controls autoplay preload="auto"></audio>
  <script>
    const ac = new AudioContext({numberOfChannels:2});
    const {
      stream
    } = msd;
    const [track] = stream.getAudioTracks();
    // track.enabled = false;
    const gainNode = new GainNode(ac, {
      gain: 1
    });
    const audio = document.querySelector("audio");
    audio.srcObject = stream;
    const res = [];
    let init = false;
    let curr = 0;
    let aw = ac.audioWorklet.addModule("audioWorklet.js").then(_ => {
      const aw = new AudioWorkletNode(ac, "audio-data-worklet-stream", {
        numberOfOutputs:2,
        processorOptions: {
          buffers: []
        }
      });
      aw.port.addEventListener("message", async e => {
        if (e.data === "stop") {
          audioTrack.stop();
          await ac.close();
        }
      });
      gainNode.connect(msd);
      aw.connect(gainNode);
      gainNode.connect(ac.destination);
      return aw;
    });
    async function processStream({
      value, done
    }) {
      try {

        if (done) {
          return this.close;
        }
        for (let i = 0; i < value.byteLength; i++) {
          res[res.length] = value[i];
        }
        let ab = void 0;
        if (init === false && (ab = await ac.decodeAudioData(new Uint8Array(res).buffer).catch(_ => void 0))) {
          init = true;
          aw = await aw;
          let [channel0, channel1] = [ab.getChannelData(0), ab.getChannelData(1)];

          aw.port.postMessage({channel0, channel1}, [channel0.buffer, channel1.buffer])
          return this.read().then(processStream.bind(this)).catch(e => {
            throw e
          })

        } else {
          // this is the problematic part
          // we have to read the entire res array and create a new Uint8Array here
          // or try and not succeed using OfflineAudioContext with start(ac.currentTime, curr)
          // and startRendering(), where we should be able to simply read value and prepend header
          // either static generic header or original header from first initial value of reader.read()
          return this.read().then(processStream.bind(this)).catch(e => {
            throw e
          })
       }
      } catch (e) {
        console.log(e);
      }
    }
    fetch("https://fetch-stream-audio.anthum.com/72kbps/opus/house--64kbs.opus?cacheBust=1")
      .then(response => {
        len = response.headers.get("content-length");
        return response.body.getReader()
      })
      .then(reader => {
        reader.read.call(reader).then(processStream.bind(reader))
      })
  </script>

AudioWorkletProcessor

class AudioDataWorkletStream extends AudioWorkletProcessor {
  constructor(options) {
    super();
    if (options.processorOptions) {
      Object.assign(this, options.processorOptions);
    }

    this.port.onmessage = e => {
      const {
        channel0, channel1
      } = e.data;
      // ideally we push to an array, e.g., this.buffers, and use async generator 
      // to run next values when curren
      // values have been processed, though for an unknown reason "does not work"
      this.len = channel0.byteLength;
      this.data = [channel0, channel1];
      this.i = 0;
    }
  }
  process(inputs, outputs) {
    const input = inputs[0];
    const [output0, output1] = outputs;
    const inputChannel = input[0];
    const [[outputChannel0], [outputChannel1]] = [output0, output1];
    if (this.data !== undefined && this.data instanceof Array && this.i < this.len) {
      // this does play the first ~1 second of the streame audio
      for (let n = 0; n < 128; this.i++, n++) {
        if (this.i >= this.len) {
          // postMessage waits until all other tasks are completed
          this.port.postMessage({currentTime, currentFrame});
          this.data = void 0;
          this.i = this.len = 0;
          return false;
        };
        outputChannel0.set([this.data[0][this.i]], n, n + 1);
        outputChannel1.set([this.data[1][this.i]], n, n + 1);
      }
      return true;
    } else {
      // stream silence if no audio input
      for (let n = 0; n < outputChannel0.length; n++) {
       outputChannel0.set(inputChannel);
      }
      return true;
    }
  }
}
registerProcessor("audio-data-worklet-stream", AudioDataWorkletStream);

[guest271314]

@hillct

We used the same approach to achieve streaming (partial download) segmented decoding of MP3 files

Can you post a link to the code?

[padenot]

Please have this discussion elsewhere, thanks.

[guest271314]

How is this not the venue for such a discussion relating to Web Audio API decodeAudioData() specification and implementation and potential workarounds for deficiencies?

[padenot]

New feature requests go in the v2 repo. New feature requests about audio decoding and encoding go in the Web Codecs repo, but this is use-case has been handled from day one there.

[guest271314]

That is on the edge of confusing. WebCodecs is not operational. There is not feature to request there because nothing is up and running and deployed at all, yet. Does WebCodecs supplant the independence of Web Audio API?

The use case has not been handled, else we would already be able to decode partial content. And memory usage associated with decodeAudioData() would not be crashing the tab.

This is about actually trying to fix decodeAudioData() not deferring to an as yet non-existent API while the pre-existing issue with decodeAudioData() persists, irrespective of any other API.

[sahi1422]

I still don't understand how to decode partial audio data. Can someone help?
Thanks.

[rtoy]

I still don't understand how to decode partial audio data. Can someone help?

You can't with decodeAudioData directly. It expects the entire file and expects to decode all of it.

You will have to do it yourself. WebCodecs will help here, I think.