Web-audio-api: AudioFrequencyWorkerNode for working in the Frequency Domain

[hughrawlinson]

At the moment, ScriptProcessorNodes and AudioWorkers are operating on the time domain buffer data. At the Web Audio Conference, it seems like there's demand for frequency domain data inside a callback that's going to get called for every audio frame.

We're thus proposing an new node called an AudioFrequencyWorkerNode, which gives the developer the option to obtain audio data, perform processing, and output in the frequency domain. This involves passing an options object to the createAudioFrequencyWorker method, specifying the input and output types.

Defaults

The AudioFrequencyWorkerNode should allow access to time domain and frequency domain data concurrently. If no options object is passed to createAudioFrequencyWorker, the input type would default to the amplitude/phase pair, as would the output. However, the options object would allow the user to choose between amplitude/phase, real/imaginary, and time domain data. The dataOut would default to the same as the dataIn, but could be set to a different data type, in case the user wants to read real/imaginary pairs in, and write out to the time domain for example.

Proposed processing structure of the AudioFrequencyWorkerNode

INPUT (time-domain)
↓
windowing
↓
FFT
↓
~ ~ ~ ~ ~ ~ ~ ~ ~
dataIn
↓
onaudioprocess
↓
dataOut
~ ~ ~ ~ ~ ~ ~ ~ ~
↓
mirror
↓
complete data
↓
IFFT
↓
windowing
↓
OUTPUT (time-domain)

Example code:

_main JS_

var aw = createAudioFrequencyWorker("worker.js",{
        dataIn:[
            "amplitude",
            "phase",
            "real",
            "imaginary",
            "signal"
        ],
        dataOut:"complex" || "amplitude" || "phase" || "signal",
        bufferSize:[power of two],
        hopSize:N,
        windowingType,
        zeroPadding:N
    }
// Signal (time domain) would be the default I/O for the AudioFrequencyWorker.
// indicating another datatype would actuate an fft and/or ifft around the user code.

_AudioWorker JS_

// callback to AudioFrequencyWorkerNode
onaudioprocess = function (e) {
  // e.amplitude[0][channel];
  // e.phase[0][channel];
  // e.real[0][channel];
  // e.imaginary[0][channel];
  // e.signal[0][channel]; // replacing the 'input', time domain.

  // edit the arrays in place, rather than having a separate output array to copy into

  for (var channel = 0; channel<e.amplitude.length;channel++){
    for(var i = 0; i < e.amplitude.length;i++){
        e.amplitude[i][channel] = e.amplitude[i][channel]>0.5 ? e.amplitude[i][channel] : 0;
    }
  }
};

Use Cases

• Feature Extraction,
• Phase Vocoding,
• Pitch Shifting,
• Pitch Track,
• Time stretching,
• Frequency domain gating effects

Jesse Allison, Hugh Rawlinson, Jakub Fiala, Nevo Segal
@jesseallison, @hughrawlinson, @jakubfiala, @nevosegal

Related

248

262

[jesseallison]

:+1:

[jakubfiala]

:8ball:

[nevosegal]

:+1:

[cristiano-belloni]

+10^5

[jesseallison]

An FFT Bin Modulation Example:

A common audio process of changing the gain on any bin. These could be done through a array of bin modulation values, processed as a whole or on a bin by bin basis.

main JS:

var bufferSize = 2048;
var binCount = bufferSize/2;

var binModNode = createAudioFrequencyWorker("binAmplitudeModulator.js",{
        dataIn: "amplitude",
        dataOut: "signal",
                bufferSize: bufferSize
    }

// binModNode.frequencyBinScalingArray[];  // Assuming parameters could be accessible directly upon instantiation... this may have to be declared.
var cutoffBin = 22;

// creating a cutoff frequency
for(var i=0; i < binCount;i++){
    var scale = (i<cutoffBin) ? 0 : 1;
    binModNode.frequencyBinScalingArray[i] = scale;
}

// adjusting a single bin
binModNode.frequencyBinScalingArray[5] = 2.5;

AudioWorker JS:

// callback to AudioWorkerNode
onaudioprocess = function (e) {
    var channelCount = e.amplitude[0].length;
  var bufferLength = e.amplitude.length;        // This could possibly be automatically generated in the AudioFrequencyWorkerNode

  var frequencyBinScalingArray = e.parameters.frequencyBinScalingArray;

  for (var channel = 0; channel<channelCount;channel++){
    for(var i = 0; i < bufferLength;i++){
        e.amplitude[i][channel] = e.amplitude[i][channel] * frequencyBinScalingArray[i];
    }
  }
};

[svgeesus]

Excellent. I called for the folks who wanted this to write it up at todays Audio WG panel at WAC, and here it is the same day. Ws cool to see it develop on the piratepad too.

[rtoy]

I fail to see what this node provides that an AudioWorker node does not.

I also find it confusing if the AudioFrequencyWorkerNode outputs a
frequency domain signal and is then connected to other nodes. Do you then
just get random garbage in and out?

I think this also raises the question of what is WebAudio? Is it intended
to be a general purpose signal processing package where you can do whatever
you want? (Modulo AudioWorkers, where you can do whatever you want, as long
as the output is an audio signal. Yes, you can abuse that too, but that's
not the intent.)

On Wed, Jan 28, 2015 at 10:41 AM, Nantonos [email protected] wrote:

Excellent. I called for the folks who wanted this to write it up at todays
Audio WG panel at WAC, and here it is the same day. Ws cool to see it
develop on the piratepad too.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71890205
.

Ray

[hughrawlinson]

Hi @rtoy,

The AudioFrequencyWorkerNode provides a lot of functionality not available in the AudioWorkerNode. It allows programmers (and composers, game designers, etc. etc. etc.) the ability to manipulate data in the frequency domain without all the prerequisite DSP knowledge necessary to accomplish similar tasks in the time domain.

In response to your second point, I should clarify the processing structure of the AudioFrequencyWorkerNode. The node acts like any other node in that it both accepts input and provides output in the time domain. If you look at the little ascii diagram in the original issue, 'INPUT' and 'OUTPUT' refer to the input and outputs of the node while 'dataIn' and 'dataOut' are the frequency domain arrays that are accessible inside the callback of the AudioFrequencyWorkerNode. The node would take care of the transformation between time and frequency both on the way in to the callback and on the way out of it.

I'm not quite sure what you're getting at with your third point, would you mind clarifying?

[hoch]

all the prerequisite DSP knowledge

I guess the prerequisite here is being able to write FFT/IFFT in AudioWorker. Besides, you still need to have DSP knowledge to manipulate mag/phase properly - doing FFT/IFFT by native code doesn't necessarily reduce the amount of knowledge you need.

Here are more nitpicks:
1) Are you proposing this because you assume FFT/IFFT with JS in AudioWorker will be too slow for realtime applications?
2) Probably browser vendors will not have the identical FFT/IFFT implementation, so what you will have in the node after FFT might vary on different browsers or platforms. (different codec or etc.) Are you okay with that? Doing FFT/IFFT with optimized JS code in AudioWorker will not have that kind of problem.

[adelespinasse]

How about, instead of this, just provide a good set of general-purpose DSP operations for Float32Array, including FFT, inverse FFT, windowing, etc.? That would let you do everything this proposal lets you do (unless I'm missing something), and also lots of other stuff.

[hughrawlinson]

There are other reasons that the AudioWorkerNode isn't suited to dealing with frequency domain data. The buffer size is set at 128 samples, which when converted to the frequency domain gives you bins that are ~345hz wide (if my maths are right). Not exactly ideal. This is to fulfil the design goal that the AudioWorkerNode doesn't introduce any latency into the audio graph. For many purposes in the frequency domain, this isn't ideal.

Probably browser vendors will not have the identical FFT/IFFT implementation

That sounds like something that should be specified... I'm very surprised to see that it seems kind of ambiguous in the spec for the AnalyserNode...

A good set of general-purpose DSP operations is being proposed, in the Web Array Math API whose status I don't know or understand.

[rtoy]

On Wed, Jan 28, 2015 at 3:23 PM, Hugh Rawlinson [email protected]
wrote:

There are other reasons that the AudioWorkerNode isn't suited to dealing
with frequency domain data. The buffer size is set at 128 samples, which
when converted to the frequency domain gives you bins that are ~345hz wide
(if my maths are right). Not exactly ideal. This is to fulfil the design
goal that the AudioWorkerNode doesn't introduce any latency into the audio
graph. For many purposes in the frequency domain, this isn't ideal.

​You have no choice on the buffer size. Nodes always get blocks of 128
frames. You have have to buffer internally if you want to process on larger
chunks. (This buffering is kind of hidden for ScriptProcessorNodes where
the buffering is done for you and the node gets larger buffers all at
once.)​

Probably browser vendors will not have the identical FFT/IFFT
implementation

That sounds like something that should be specified... I'm very surprised
to see that it seems kind of ambiguous in the spec for the AnalyserNode
http://webaudio.github.io/web-audio-api/#the-analysernode-interface...

​Oops. That's a bug and we should specify precisely what the FFT is. (I
know of at least 2 ways of defining the forward part, and at least 3 ways
to scale it.)​

I'm pretty sure, however, that currently everyone does it the same way, and
any differences are rounding errors depending on the exact FFT algorithm
used.

A good set of general-purpose DSP operations is being proposed, in the Web
Array Math API http://opera-mage.github.io/webarraymath/ whose status I
don't know or understand.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71938414
.

Ray

[rtoy]

On Wed, Jan 28, 2015 at 2:34 PM, Hugh Rawlinson [email protected]
wrote:

Hi @rtoy https://github.com/rtoy,

The AudioFrequencyWorkerNode provides a lot of functionality not available
in the AudioWorkerNode. It allows programmers (and composers, game
designers, etc. etc. etc.) the ability to manipulate data in the frequency
domain without all the prerequisite DSP knowledge necessary to accomplish
similar tasks in the time domain.

​I think that if you're manipulating things in the frequency domain, you
have a fair amount of DSP knowledge already. It's easy enough for an
AudioWorkerNode to do an FFT internally using any of the available JS FFT
libraries out there.​

In response to your second point, I should clarify the processing
structure of the AudioFrequencyWorkerNode. The node acts like any other
node in that it both accepts input and provides output in the time domain.
If you look at the little ascii diagram in the original issue, 'INPUT' and
'OUTPUT' refer to the input and outputs of the node while 'dataIn' and
'dataOut' are the frequency domain arrays that are accessible inside the
callback of the AudioFrequencyWorkerNode. The node would take care of the
transformation between time and frequency both on the way in to the
callback and on the way out of it.

​Ah, thanks for the clarification. Audio in and out makes much more sense.​

I'm not quite sure what you're getting at with your third point, would you
mind clarifying?

​Basically, what kind of native nodes should WebAudio supply? ​

​Is the intent to supply a huge set of nodes where you can do just about
any general purpose DSP technique as if you were using, say, Matlab-like
clone running in a browser? I don't think that's the goal.​

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71931353
.

Ray

[rtoy]

Aren't there JS libraries out there already to do these kinds of things? I
don't see that this falls under WebAudio's goals.

On Wed, Jan 28, 2015 at 3:04 PM, Alan deLespinasse <[email protected]

wrote:

How about, instead of this, just provide a good set of general-purpose DSP
operations for Float32Array, including FFT, inverse FFT, windowing, etc.?
That would let you do everything this proposal lets you do (unless I'm
missing something), and also lots of other stuff.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71935694
.

Ray

[sebpiq]

​I think that if you're manipulating things in the frequency domain, you
have a fair amount of DSP knowledge already

With that reasoning most of the Web Audio API doesn't make sense does it ;)

Even with fair knowledge of DSP, there is a some not-completely trivial
stuff there such as overlapping windows and so on.

On Thu, Jan 29, 2015 at 1:00 AM, rtoy [email protected] wrote:

Aren't there JS libraries out there already to do these kinds of things? I
don't see that this falls under WebAudio's goals.

On Wed, Jan 28, 2015 at 3:04 PM, Alan deLespinasse <
[email protected]

wrote:

How about, instead of this, just provide a good set of general-purpose
DSP
operations for Float32Array, including FFT, inverse FFT, windowing,
etc.?
That would let you do everything this proposal lets you do (unless I'm
missing something), and also lots of other stuff.

—
Reply to this email directly or view it on GitHub
<
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71935694>

.

Ray

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71943104
.

_Sébastien Piquemal_

-----* @sebpiq*
----- http://github.com/sebpiq
----- http://funktion.fm

[hoch]

How about, instead of this, just provide a good set of general-purpose DSP operations for Float32Array, including FFT, inverse FFT, windowing, etc.?

A good set of general-purpose DSP operations is being proposed, in the Web Array Math API whose status I don't know or understand.

@adelespinasse @hughrawlinson That seems to be a nice complement for AudioWorker.

[sebpiq]

Yes... web array math API sounded great but I haven't heard any news for a
good 6 months now :(

On Thu, Jan 29, 2015 at 1:07 AM, Hongchan Choi [email protected]
wrote:

How about, instead of this, just provide a good set of general-purpose DSP
operations for Float32Array, including FFT, inverse FFT, windowing, etc.?

A good set of general-purpose DSP operations is being proposed, in the Web
Array Math API whose status I don't know or understand.

@adelespinasse https://github.com/adelespinasse @hughrawlinson
https://github.com/hughrawlinson That seems to be a nice complement for
AudioWorker.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71943895
.

_Sébastien Piquemal_

-----* @sebpiq*
----- http://github.com/sebpiq
----- http://funktion.fm

[hughrawlinson]

I agree with @sebpiq, the ability to do something in AudioWorker doesn't negate the need for a node. You could implement every single node in the spec with AudioWorker, if that's the reasoning then why not just make people implement their own Oscillators with AudioWorker rather than supply OscillatorNode?

​Is the intent to supply a huge set of nodes where you can do just about
any general purpose DSP technique as if you were using, say, Matlab-like
clone running in a browser? I don't think that's the goal.​

I don't think you need a huge set of nodes to do any general purpose DSP technique. I do however think it's good to be able to work in both the time and frequency domains, and AudioWorker caters for the time domain, so why not have something that caters for the frequency domain? If you want to optimise for as few nodes as possible, then surely we should all be implementing our own oscillators in AudioWorkers rather than using OscillatorNodes, I don't really think having fewer nodes is a great design goal. Having general purpose nodes though, is, and I think AudioFrequencyWorker is general enough to warrant being a node.

@adelespinasse @hughrawlinson That seems to be a nice complement for AudioWorker.

Yeah, the Web Array Math API functions would definitely be useful inside of AudioWorker, but the spec doesn't seem to be progressing, and as it's a separate spec it may not be ready for years even after the Web Audio API is released.

[jakubfiala]

@rtoy So the difference between the knowledge necessary to, say, do basic spectral modulation in your AudioWorker, and performing a highly optimized Fast Fourier Transform is quite vast. The main motivation behind this proposal is really that the majority of Web Audio developers aren't able to do that easily, me being one of them. Given that it's one of the fundaments of audio DSP and pretty much everybody in the field gets to use the FFT spectrum at some point, I think it's more than fitting to have a _really damn good_, as well as _really damn fast_ version of it in WA.

Not to mention that we already _do_ have an FFT in WA, and this is just an attempt to make it useful for more than just simply visualizing the spectrum, which is pretty much all that the AnalyserNode is good for. Actually, in this sense I can take your argument even further, and say – if there already is a native FFT in Web Audio, why should we have people implementing it again in JS? Isn't that just horribly inefficient?

Mind you, we originally envisaged this as an extension to the normal AudioWorker (so no extra nodes), but after a consultation with Paul Adenot we realised it would actually only work as a separate node mainly because of the fixed buffer size constraint.

[cwilso]

If you're looking for an FFT that produces sound - a frequency-domain
transformer - then yes, obvious the Analyser is not for you.

My personal take on this is that this would likely be better served by a
separate FFT library, and someone dealing with the array aggregation in
interesting ways. (We could provide a "buffering node" that enabled
processing of large blocks with corresponding latency - but I'm not
personally convinced quite enough that it's necessary (=hard/nonperformant
enough to do yourself). At any rate, this is DEFINITELY separate from the
straight AudioWorker.

That said - I'll defer to the WG, but I don't think this is a v1 feature.
You can, for the moment, build the buffering semantic yourself in a worker,
and the FFT in JS (or use Web Array Math if there's an implementation).

On Thu, Jan 29, 2015 at 12:24 PM, Jakub Fiala [email protected]
wrote:

@rtoy https://github.com/rtoy So the difference between the knowledge
necessary to, say, do amplitude modulation in your AudioWorker, and
performing a highly optimized Fast Fourier Transform is quite vast. The
main motivation behind this proposal is really that the majority of Web
Audio developers aren't able to do that easily, me being one of them. Given
that it's one of the fundaments of audio DSP and pretty much everybody in
the field gets to use the FFT spectrum at some point, I think it's more
than fitting to have a _really damn good_, as well as _really damn fast_
version of it in WA.

Not to mention that we already _do_ have an FFT in WA, and this is just
an attempt to make it useful for more than just simply visualizing the
spectrum, which is pretty much all that the AnalyserNode is good for.
Actually, in this sense I can take your argument even further, and say – if
there already is a native FFT in Web Audio, why should we have people
implementing it again in JS? Isn't that just horribly inefficient?

Mind you, we originally envisaged this as an extension to the normal
AudioWorker (so no extra nodes), but after a consultation with Paul Adenot
we realised it would actually only work as a separate node mainly because
of the fixed buffer size constraint.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71952201
.

[chrislo]

I think @cwilso's point is a very valid one. I think the existence of AudioWorker and presumably the proliferation of libraries that will then be developed to handle the buffering/aggregation and (I)FFT maths will make it very clear to us over time what a future AudioFrequencyWorker's interface should look like.

Exposing all of the buffering parameters, as well as those of the FFT algorithm (specifying overlap, window functions, scaling etc) is going to be a hard thing to get right - I imagine a consensus on this might develop over time in user code that can be perhaps incorporated into the API later for performance / ease of use reasons.

[chrislo]

(That being said, I think it's fantastic that this issue has been raised as a direct consequence of the public Audio WG consultation meeting on Wednesday)

[notthetup]

Does anyone have any performance numbers on optimised JS implementations of FFT running inside ScriptProcessorNode along with FFT implementation optimised with asm.js + SIMD.js (and maybe also PNaCl)?

I feel knowing performance differences between native and JS implementations would be very important to understand what we're actually gaining from making this functionality into a native Node.

Ofcourse, strictly speaking we should looking at the performance in the AudioWorker (closer to native Nodes in terms of threading), but that will have to wait, so we could start with ScriptProcessorNode for now.

[cristiano-belloni]

@notthetup - A good starting point could be this library: https://github.com/corbanbrook/dsp.js/.
Although it's not asm.js optimized, the RFFT class in the lib performs a forward FFT.

A good native FFT implementation is http://www.fftw.org/, for comparison.

[cristiano-belloni]

@rtoy:

​I think that if you're manipulating things in the frequency domain, you
have a fair amount of DSP knowledge already. It's easy enough for an
AudioWorkerNode to do an FFT internally using any of the available JS FFT
libraries out there.​

Yep, but isn't it re-inventing the [possibly square] wheel? The FFT is a building block of DSP, it makes sense to have it implemented efficiently and natively, without everyone having to re-implement it, or using libraries that re-implement it.

Compare to oscillators: it's reasonably easy to implement an oscillator in any given programming language, but the Web Audio API implements them natively and efficiently, with a single interface, because they're elementary building blocks of audio processing. Why the FFT shouldn't be treated equally?

[notthetup]

@janesconference Thanks. I also found this which seems to use asm.js. https://github.com/g200kg/Fft-asm.js/commits/master

I will try to create a example to test performance when I'm back home.

[rtoy]

On Wed, Jan 28, 2015 at 4:25 PM, Hugh Rawlinson [email protected]
wrote:

I agree with @sebpiq https://github.com/sebpiq, the ability to do
something in AudioWorker doesn't negate the need for a node. You could
implement every single node in the spec with AudioWorker, if that's the
reasoning then why not just make people implement their own Oscillators
with AudioWorker rather than supply OscillatorNode?

​Perhaps if history had proceeded differently and an AudioWorker existed
from the beginning, all nodes would have been defined or implemented in
terms of an AudioWorker. That's not how history went. I'm not sure, but
it's conceivable that implementations could change existing nodes to be
AudioWorkers underneath if so desired.​

​Is the intent to supply a huge set of nodes where you can do just about
any general purpose DSP technique as if you were using, say, Matlab-like
clone running in a browser? I don't think that's the goal.​

I don't think you need a huge set of nodes to do any general purpose DSP
technique. I do however think it's good to be able to work in both the time
and frequency domains, and AudioWorker caters for the time domain, so why
not have something that caters for the frequency domain? If you want to
optimise for as few nodes as possible, then surely we should all be
implementing our own oscillators in AudioWorkers rather than using
OscillatorNodes, I don't really think having fewer nodes is a great design
goal. Having general purpose nodes though, is, and I think
AudioFrequencyWorker is general enough to warrant being a node.

​I grant that it is useful. But not beyond what an AudioWorker could do
just as well.​

​And given the fact that even with the few nodes we have today, we _still_
have failed after several years to specify them completely enough that
someone could implement them from the spec. Fewer nodes are good. :-)​

@adelespinasse https://github.com/adelespinasse @hughrawlinson
https://github.com/hughrawlinson That seems to be a nice complement for
AudioWorker.

Yeah, the Web Array Math API functions would definitely be useful inside
of AudioWorker, but the spec doesn't seem to be progressing, and as it's a
separate spec it may not be ready for years even after the Web Audio API is
released.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-71945937
.

Ray

[cristiano-belloni]

​Perhaps if history had proceeded differently and an AudioWorker existed
from the beginning, all nodes would have been defined or implemented in
terms of an AudioWorker. That's not how history went. I'm not sure, but
it's conceivable that implementations could change existing nodes to be
AudioWorkers underneath if so desired.​

That's interesting. I was under the impression that nodes were specialized, fast and memory-friendly ways to implement building blocks? Wouldn't be penalizing implementing them via a worker?
(e.g.: easily discardable AudioBufferSourceNodes)

[notthetup]

Sorry this took a while. But I have a basic performance test setup for how long it takes to run FFTs using some of the popular FFT libraries, plotted against ScriptProcessor callback times. Of course we would want to do this with the AudioWorker when it comes, but this gives us data points to start with. Also please let me know if you see any bugs.

http://chinpen.net/webaudiofftperf/

[jakubfiala]

@notthetup awesome! As far as I understand, the fft-asm.js option isn't working yet, right?

[notthetup]

@jakubfiala Yes. I haven't implemented that yet. Hopefully this weekend.

[echo66]

@notthetup , AFAIK, you currently have two options for 1D FFT, implemented using ASM.js:

https://github.com/echo66/FFTW3-emscripten
https://github.com/drom/fourier

benchmarks:

http://jsperf.com/comparing-two-fft-functions/3
http://jsperf.com/comparing-two-fft-functions/5
http://jsperf.com/benchmark-for-fftw-js-and-drom-s-fft/2

[notthetup]

@echo66 Thanks!! Will look at those

If anyone has some time (I'm really flooded for the next few weeks), please feel free to PR https://github.com/notthetup/webaudiofftperf

[echo66]

Greetings to everyone!

In the last month, I have been working in real-time time stretching and pitch shifting using a phase vocoder, using my own implementations and @janesconference PitchShifter. Currently, there seems to exist a big bottleneck in FFT calculation, resulting in audio dropouts if you have more than 4-5 nodes with time-stretching and/or pitch shifting. For me (at least), the absence of an alternative to the javascript FFT implementations is a major roadblock.

Note 1: I forgot about the pitch shifter implemented by @cwilso . That one doesn't seem to suffer from what I mentioned I the previous paragraph. Unfortunately, for me, it provides a lower (audio) quality than @janesconference implementation or my phase vocoder + resampling.
Note 2: It should be noted that the implementations I have created/experimented with do not use SIMD.

[cwilso]

The pitch shifting I did is granular resynthesis, which is going to be lower audio quality. I'm expecting a lot of the dropouts you're experiencing are because you're using ScriptProcessors for the nodes (yes?), and the thread-hopping starts thrashing.

[echo66]

Yes, I'm using ScriptProcessors, @cwilso .

[cristiano-belloni]

My pitchshifter, as far as I remember, uses the FFT implementation found here: https://github.com/corbanbrook/dsp.js/blob/master/dsp.js

I remember it did real-time shifting back in the days, but consuming a lot of CPU.
Maybe, if you replace the FFT with a faster one, the performance will be better.

[Incidentally, is this not a exemplary case where a native FFT would be very useful? You did a lot of Jsperf comparison in this thread, but what happens if we compare an asm.js implementation to a native, efficient one?]

[echo66]

I replaced by this implementation: https://github.com/drom/fourier. I can do a pull request, if you want, @janesconference .

The implementation you used, when compared with Matlab and FFTW output, it is returning strange values (I think I even opened an issue in dsp.js repo).

I agree with you that this is an exemplary use case for native FFT. And due to the historic importance of the FFT in DSP, I find it odd that it is not considered in WAA. One could say that there are other important transforms but, if that's the case, why is there an Analyzer Node that outputs the FFT magnitude? It could be implemented as a ScriptProcessor (IMHO). And why doesn't the Analyzer Node provide the FFT phase?

By the way: I don't remember the last time I saw a DSP library/framework/platform (besides WAA) without explicit support for FFT.

[rtoy]

I have never considered WebAudio as a DSP library/framework/platform. Hence, I don't miss an FFT.

[cristiano-belloni]

I can do a pull request, if you want

Yes, thanks. Pitchshifter was made years ago, and I didn't find the time to update it properly :)

[echo66]

@rtoy , then what is WAA?

[cristiano-belloni]

I have never considered WebAudio as a DSP library/framework/platform. Hence, I don't miss an FFT

Well, it offers mid-level components, such as oscillators and biquads, and fairly high-level components, like the panner and the compressor. It sure is not a library / framework / platform, but it is not a bare-bones API, either. I wonder why - and I say that with sincere curiosity - a compressor is OK but a Fourier transform is too much.

[adelespinasse]

Things like FFT don't belong in the Web Audio API because they aren't
specific to audio and don't need to be coupled with an AudioContext.
They're just operations on arrays. It would be really nice if there were a
standard library of fast vector operations (last I knew there wasn't even a
way to add two Float32Arrays without writing a loop in JavaScript), and I
would strongly favor having some transforms in that (especially since
there's already FFT code in each browser), but they don't belong in the Web
Audio API.

This is the fundamental reason why the original proposal of having an
AudioFrequencyWorkerNode struck me as API bloat. It's much better to have
orthogonal APIs that can be combined in arbitrary ways.

On Wed, May 20, 2015 at 6:32 PM, Cristiano Belloni <[email protected]

wrote:

I have never considered WebAudio as a DSP library/framework/platform.
Hence, I don't miss an FFT

Well, it offers mid-level components, such as oscillators and biquads, and
fairly high-level components, like the panner and the compressor. It sure
is not a library / framework / platform, but it is not a bare-bones API,
either. I wonder why - and I say that with sincere curiosity - a compressor
is OK but a Fourier transform is too much.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104062614
.

[notthetup]

SIMD.js is should help with optimising the vector operation. But we'll have to wait till ES7 for that to be standardised, although parts of the implementation are already in FF-Nightly and Chrome-Nightly.

https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/2PIOEJG_aYY

[echo66]

@adelespinasse , I understand your point. Still, if that is the case, why is there an AnalyzerNode? And why is it a FFT instead, idk, a wavelet transform (the fourier transform is a special case of the wavelet transform, AFAIK)? Since I'm not part of the WAA steering panel, I'm just trying to get the reason why you chose to create the Analyser Node (and why only the magnitude is being reported) instead of standardizing an FFT API. If it is just for spectrum rendering, we could just use a javascript FFT implementation inside a ScriptProcessor (after all, we still need a ScriptProcessor to render the spectrum).

[artofmus]

The problem is that it is impossible implement FFT WINDOWING in web audio. For this it is necessary DelayNode could be less than one buffer (for example 1 sample), or need to work with several buffers in ScriptProcessorNode (present and previous) for overlaping. Or I'm wrong? Tell me how you solved the problem of overlap windows in AudioFrequencyWorkerNode?

[echo66]

@hughrawlinson , I think @artofmus is talking to you.

[echo66]

@artofmus , is there an issue, for you, to use a ScriptProcessor/AudioWorker as buffer reader, which performs windowing and takes care of the hop sizes?

[artofmus]

@echo66 so, yes. And what are the alternatives ScriptProcessor/AudioWorker for this?

[echo66]

If you are reading an AudioBuffer, you can use a ScriptProcessor/AudioWork to iterate through the samples. Take a look at this code excerpt:

node.onaudioprocess = function (e) {

    var il = buffer.getChannelData(0);
    var ir = buffer.getChannelData(1);

    var ol = e.outputBuffer.getChannelData(0);
    var or = e.outputBuffer.getChannelData(1);

    // Fill output buffers (left & right) until the system has enough processed samples to reproduce.
    do {

        var buf1 = il.subarray(position, position+BUFFER_SIZE);
        var buf2 = ir.subarray(position, position+BUFFER_SIZE);

        position += phasevocoder1.get_analysis_hop();

        // Process left input channel
        outBuffer1 = outBuffer1.concat(phasevocoder1.process(buf1));

        // Process right input channel
        outBuffer2 = outBuffer2.concat(phasevocoder2.process(buf2));

    } while(outBuffer1.length < BUFFER_SIZE);

    ol.set(outBuffer1.splice(0, BUFFER_SIZE));
    or.set(outBuffer2.splice(0, BUFFER_SIZE));

}

At each execution of the onaudioprocess method, I'm processing several frames, each one overlapping the previous one(s) (phasevocoder1.get_analysis_hop()). By using a ScriptProcessor, I can (manually) control the frame hopping at the input. To control it at the output, you need an intermediary array/buffer.

[artofmus]

@echo66 It is clear, let's check how it works. And do not tell me which code you use to calculate FFT IFFT and operations with complex numbers?

[echo66]

Sure! I used this one: https://github.com/drom/fourier

[echo66]

One tip: if you can avoid allocating new arrays during during the processing, you can avoid certain issues with the garbage collector. In the example I have posted in here, I'm using Array.concat and Array.splice. And, AFAIK, this requests the creation of new arrays.

[cristiano-belloni]

Things like FFT don't belong in the Web Audio API because they aren't
specific to audio and don't need to be coupled with an AudioContext.
They're just operations on arrays.

But aren't filters "not specific to audio" and "just operations on arrays"? And convolution? And any other DSP technique?

The problem seems constantly shifting: sometimes it's 'they should not be owned by WAA' (but we have filters and convolvers and waveshapers, which are _not_ exclusive to the audio domain), sometimes it's 'there are already implementations in Javascript' (but they're considerably slower, and the same can be said for filters and convolvers, no?), sometimes it is 'they change domain, what will we do when we connect those nodes to other nodes?' (while they would probably be another type of audioworker, and what an AW spits out is complete responsibility of the user).

I really fail to see what's wrong in audio developers asking for native FFT.

It's because FFT in Javascript is generally slow and we don't generally use it, because when we do, it feels we're stuck in '97 with Cubase 3.5 VST, when phase-vocoding made the [tab] CPU rise to 90%, and we thought we solved that particular problem 20 years ago.

[echo66]

@janesconference , +9001.

@janesconference , do you know any (even if remote) reason why only the AnalyserNode doesn't have the phase? Seriously, since I can remember, I never got that design choice. I'm really really curious about that.

[artofmus]

@echo66 @janesconference, I fully support !

[adelespinasse]

@janesconference, I would not say that filters/convolution are just
operations on arrays; they're operations on infinite streams of numbers.
Not quite the same thing as being inherently audio, but much closer to it
than FFTs are.

BUT, I agree with everything else you said. It's frustrating waiting for
the Web platform to catch up with the rest of the world. And my point (that
FFT should be in a different API) doesn't help with that, since dealing
with more than one committee is even worse than dealing with one.

@echo66, I think the AnalyserNode is the way it is (including the odd
spelling) because, pre-standardization, someone (Chris Rogers?) thought it
would be cool to be able to make "spectrum analyzer" types of
visualizations, and made it happen. Committees slow things down (hence our
frustration), but without some kind of review process, you end up with
strange decisions like that.

(And I'm not on the committee either, although I'm strongly considering
crashing the face-to-face meeting in 2 weeks, since it happens to be around
the block from my home.)

On Thu, May 21, 2015 at 9:27 AM, artofmus [email protected] wrote:

@echo66 https://github.com/echo66 @janesconference
https://github.com/janesconference, I fully support !

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104277261
.

[echo66]

@adelespinasse , thanks for the sincere opinion! :)

Well, now I can "sense the pain" that one my thesis supervisors mention when talking about Semantic Web committees.

[jakubfiala]

I think it's necessary to reiterate here that the lack of phase spectrum output isn't the only issue with AnalyserNode – another one is the fact that we have no way of knowing which part of the audio stream corresponds to the current magnitude data from AN. Even though you can get the synchronized time domain buffer from it, you would have to then figure out where that particular buffer belongs in the source to do any kind of further work with it. You'd also still be interfacing with the node from the JS thread.

This restricts the usability of the node to visualisation of the frequency spectrum, which IMO offers too few possibilities to justify it being a separate node. In fact, I'd totally understand if _this node_ was a JS library.

Another option would be to simply upgrade the AnalyserNode, by giving it an onaudioprocess event handler, and exposing the FFT data within that scope. However, that does sound a lot like a freq domain version of the ScriptProcessor, and I have a feeling nobody really likes the ScriptProcessor :)

[jesseallison]

I find it interesting that there is a need to defend to the audio programming community that the FFT is important for the WebAudio API. To address a couple of points brought up:

The FFT is an operation on an infinite stream of numbers as well. It happens to create an array from a portion of that stream, but that array is itself a stream of arrays, they just cover a longer time span than a single sample. In the end, the FFT allows users to deal with sound as a spectrum instead of just streams of samples. This is perhaps more inherently audio than a stream of samples.

Also, all of the processors that we have mentioned are not inherently audio - filtering, convolution (time-domain or frequency-domain), modulation, are used anywhere waves exist (or streams of numbers for that matter), from coastal modeling to gravitational and electromagnetic waves -- It's all the same DSP. The point is, it can be used for audio and should be available through the WebAudio API if we're trying to enable a flexible audio programming environment in the browser.

At any rate, here's my reasoning for including it. Because the FFT/iFFT is a computationally expensive process and has the difficult programming problem of overlapping frames, windowing, etc., having a built in way to handle it will enable frequency-domain processing code to develop that isn't encumbered by the specifics of getting into and out of the frequency domain. It should be noted that most (if not all? Supercollider, Pd, Max, Chuck, Jamoma, CSound, ...) audio programming language/frameworks out there provide exactly this: an FFT and iFFT object (code/function/whatever) that allows you to work in the frequency domain without the details of the transform. Also working with audio in the frequency domain is only growing in it's importance in DSP, audio analysis, synthesis, etc.

I'm also a fan of getting a wavelet transform in the API, but it isn't used in the audio realm much yet. It's day will come...

[cristiano-belloni]

Thanks for the corrections guys, but the FFT (fast fourier transform) is just the algorithm to implement the DFT (discrete fourier transform), that operates on finite lists of samples, and spits out a finite series of complex sinusoids, at least as far as I can remember.

Regarding splitting WAA and having an hypothetical DSP API, I completely agree.
But, most of all, it would be nice to have a way to do efficient DSP inside the browser. Currently, whoever wants to do serious signal processing is forced to go native.

If we want the open web to compete with apps (at least in the audio niche), we should have a way to do processing at a comparable rate of efficiency.

[rtoy]

On Thu, May 21, 2015 at 9:41 AM, Cristiano Belloni <[email protected]

wrote:

Thanks for the corrections guys, but the FFT (fast fourier transform) is
just the algorithm to implement the DFT (discrete fourier transform), that
operates on finite lists of samples, and spits out a finite series of
complex sinusoids, at least as far as I can remember.

Regarding splitting WAA and having an hypothetical DSP API, I completely
agree.
But, most of all, it would be nice to have a way to do efficient DSP
inside the browser. Currently, whoever wants to do serious signal
processing is forced to go native.

​I don't think that's WebAudio's fault. Personally, I would go talk to my
favorite Javascript implementor and encourage him to make the compiler do a
better job compiling up DSP code. Then everybody wins everywhere instead
of this one narrow case in WebAudio.​

If we want the open web to compete with apps (at least in the audio
niche), we should have a way to do processing at a comparable rate of
efficiency.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104348509
.

Ray

[cristiano-belloni]

​I don't think that's WebAudio's fault. Personally, I would go talk to my
favorite Javascript implementor and encourage him to make the compiler do a
better job compiling up DSP code. Then everybody wins everywhere instead
of this one narrow case in WebAudio.​

This is not a narrow case in Web Audio. This is having native DSP in the browser, which is comparatively easier and, most of all, more immediate, than rewriting V8 to make it reach near-native speeds when it comes to DSP. Computationally optimize a javascript compiler to perform comparably to optimized C/C++ code in a cpu-bound operational subset seems far more irrealistical than hooking FFT code to a worker, exactly like you did for all the other Audionodes.

Or maybe I'm wrong, and we should do like you say: next stop, we'll rewrite libjingle in Javascript and bully compiler vendors to make it go faster when it comes to suppress echo in realtime communications.

Maybe the miscommunication, here, is that you're talking at a philosophical level and I'm talking about what we can do now (or in the immediate future), pragmatically, with the open Web.

I still don't understand why we have a freaking convolver, a sophisticated 3d panner, an analyzer node (that most probably uses an FFT implementation internally), but when it comes to have frequency-domain workers, it's a big horrified no-no.

[artofmus]

When I learned about web audio, I was happy because the work with sound in JavaScript has become almost as simple and understandable for musicians who have something to do with programming like MAX MSP PD, etc., but the lack of nodes to work with the spectrum with native optimized windowing etc. a little disappointing..( I would like to know this will be in the future or principle not?

[artofmus]

@echo66 When i using fourier.js observed very very high CPU perf. and instability work of browser. Whether I use the library to compute? ifft for the same only fourier.idft. code:

var processor = audioContext.createScriptProcessor(1024, 2, 2); 

processor.onaudioprocess = function(audioProcessingEvent) {     
  var inputBuffer = audioProcessingEvent.inputBuffer;
  var outputBuffer = audioProcessingEvent.outputBuffer;
  var inputDataL = inputBuffer.getChannelData(0);
  var inputDataR = inputBuffer.getChannelData(1);
  var outputDataL = outputBuffer.getChannelData(0);
  var outputDataR = outputBuffer.getChannelData(1);

  var fft = fourier.dft(inputDataL,inputDataR);       

  for (var sample = 0; sample < inputBuffer.length; sample++) {
    outputDataL[sample] = fft[0][sample]; 
    outputDataR[sample] = fft[1][sample];       
  }   
}

[rtoy]

On Thu, May 21, 2015 at 10:33 AM, Cristiano Belloni <
[email protected]> wrote:

​I don't think that's WebAudio's fault. Personally, I would go talk to my
favorite Javascript implementor and encourage him to make the compiler do a
better job compiling up DSP code. Then everybody wins everywhere instead
of this one narrow case in WebAudio.​

This is not a narrow case in Web Audio. This is having native DSP in the
browser, which is comparatively easier and, most of all, more immediate,
than rewriting V8 to make it reach near-native speeds when it comes to DSP.
Computationally optimize a javascript compiler to perform comparably to
optimized C/C++ code in a cpu-bound operational subset seems far more
irrealistical than hooking FFT code to a worker, exactly like you did for
all the other Audionodes.

Or maybe I'm wrong, and we should do like you say: next stop, we'll
rewrite libjingle in Javascript and bully compiler vendors to make it go
faster when it comes to suppress echo in realtime communications.

Maybe the miscommunication is that you're talking at a philosophical level
and I'm talking about what we can do now (or in the immediate future),
pragmatically, with the open Web

I still don't understand why we have a freaking convolver, a sophisticated
3d panner, an analyzer node (that most probably uses an FFT implementation
internally), but when it comes to have frequency-domain workers, it's a big
horrified no-no.

​I guess it's a difference in philosophy. As I said, I never considered
WebAudio as a general DSP framework, so missing some DSP stuff isn't a show
stopper for me. If you think WebAudio is a DSP framework, then, yeah, it's
missing tons of stuff and includes lots of strange things. And certainly
that philosophy colors your expectations.​

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104365958
.

Ray

[andremichelle]

Hi all!

I actually like the idea of an AudioFrequencyWorkerNode but totally agree that FFT should be provided natively in general in order to get the most performance out of it. FFT is interesting for a bunch of things, not only audio processing. So in best case scenario you will get such a node natively fast by writing just a few lines of javascript code under your control. There are a lot of things to consider as the window-size, window-shifting and window-shape. This should be all in your hands and might blow up the initial idea/practicability from scratch.

I personally think and hope that the web-audio-api is considered as a vision for the future. It is currently a rather simple api with a bunch of modules. A web-audio-api should be more than just a pure audio processing chain. It should enable everybody to create playful things but also very complex application up to a DAW. Therefore you need to be able to divide the current audio-block globally to enable smooth looping inside your arrangement. You need to be able to have nodes inside the audio-thread that distributes events (sequencing) like notes and automation to your audio-synthesis nodes. In the end it is all about a proper graph to solve, which is already done. It should not be the task of the ui-thread to provide events, unless it are ui-events like turning a know etc.. The big picture is then to promote it to different platforms as Android and IOS, trying to make it a standard for everybody to use. A man can dream :)

For Audiotool, we would love to re-create all devices within the web-audio-api, but we are targeting different platforms in future. So for now we are sticked to create everything inside a single ScriptProcessor, not getting the benefit of multi-threading but looking forward for AudioWorker, where we still only can use one node since it lack the important features mentioned above.

This message probably worths a new subject but I was following the AudioFrequencyWorkerNode idea for a while and it inspired me now for that conclusion :)

André Michelle
http://www.audiotool.com http://www.audiotool.com/

On 21 May 2015, at 20:51, rtoy [email protected] wrote:

On Thu, May 21, 2015 at 10:33 AM, Cristiano Belloni <
[email protected]> wrote:

​I don't think that's WebAudio's fault. Personally, I would go talk to my
favorite Javascript implementor and encourage him to make the compiler do a
better job compiling up DSP code. Then everybody wins everywhere instead
of this one narrow case in WebAudio.​

This is not a narrow case in Web Audio. This is having native DSP in the
browser, which is comparatively easier and, most of all, more immediate,
than rewriting V8 to make it reach near-native speeds when it comes to DSP.
Computationally optimize a javascript compiler to perform comparably to
optimized C/C++ code in a cpu-bound operational subset seems far more
irrealistical than hooking FFT code to a worker, exactly like you did for
all the other Audionodes.

Or maybe I'm wrong, and we should do like you say: next stop, we'll
rewrite libjingle in Javascript and bully compiler vendors to make it go
faster when it comes to suppress echo in realtime communications.

Maybe the miscommunication is that you're talking at a philosophical level
and I'm talking about what we can do now (or in the immediate future),
pragmatically, with the open Web

I still don't understand why we have a freaking convolver, a sophisticated
3d panner, an analyzer node (that most probably uses an FFT implementation
internally), but when it comes to have frequency-domain workers, it's a big
horrified no-no.

​I guess it's a difference in philosophy. As I said, I never considered
WebAudio as a general DSP framework, so missing some DSP stuff isn't a show
stopper for me. If you think WebAudio is a DSP framework, then, yeah, it's
missing tons of stuff and includes lots of strange things. And certainly
that philosophy colors your expectations.​

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104365958
.

Ray
—
Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-104386050.

[echo66]

Heey, Andre! First of all, great work on Audiotool! You and I have a similar dream. :)

By the way, due to this major issue with WAA, Im looking into ActionScript alternatives. 10 minutes ago, I just found out about one of your audio frameworks for AS. Tomorrow, I will try AS for the first time. Im kind of tired of this issue with FFT and WAA. Afterall, I have a thesis prototype to implement.

[cwilso]

So, to add my two cents: Web Audio API v 1 was never intended to be a general DSP framework for every signal processing function you might choose to do - EXCEPT through scriptprocessor (now audioworker). It _was_, in Chris Roger's original design, intended to provide convenient nodes for common audio processing tasks - like low-pass filtering, for example, with easy-to-grok frequency and Q parameters. That's not a replacement for arbitrary FFT processing - but it does hit the necessary scenarios and is far easier for far more developers to understand. Yes, we may need a general FFT node; I'm not against someone making a proposal for that - but the first place to start would be with an FFT library in JS, hooked in via audioworker. That would probably help inform the design.

@janesconference I'll point out you have a Convolver because room effects like reverb are super-common; the super-sophistication of Panner I think was a mistake, personally, and that's why I've pushed for StereoPanner in the spec, but everybody also wants to pan; there's a (relatively simplistic) Analyser because freakin' EVERYBODY wants music/audio visualization, and it's (compared to doing your own FFT when you're not a DSP expert) very easy to use. And dynamics compression is in there because it's very easy to mess up and overdrive your output signal, and digital clipping sucks.

@andremichelle note that AudioWorker will not give you "the benefit of multi-threading", per se - it just gives you the benefit of keeping your audio code out of the main UI thread. If that's what you meant, great.

[cristiano-belloni]

I'll point out you have a Convolver because room effects like reverb are super-common; the super-sophistication of Panner I think was a mistake, personally, and that's why I've pushed for StereoPanner in the spec, but everybody also wants to pan; there's a (relatively simplistic) Analyser because freakin' EVERYBODY wants music/audio visualization, and it's (compared to doing your own FFT when you're not a DSP expert) very easy to use. And dynamics compression is in there because it's very easy to mess up and overdrive your output signal, and digital clipping sucks.

And I personally love all of them, even the 3d panner that always shifts my output samples in wav.hya.io. Compared to the old times of the now-defunct FF AudioData API (I think it was called like that), WAA is super easy to use and provides native, efficient ways to use DSP building blocks.

The first version of my phase vocoding pitchshifter was for Audio Data, in Firefox 4.0, I think, and it had been a nightmare to get right. I guess that the bottom line of all this (AD dead, WAA alive and well) is that sometimes you have to go native, and I wouldn't go back to pulling samples and getting 100% CPU after chaining 3 real time effects.

That said, I guess that, like EVERYBODY wants efficient and cool 3d panning, impulse convolving and easy filtering, there's a good percent of all those everybodies that would like to mess with the spectral frequency and do vocoding, pitch and time shifting, frequency-domain filtering, audio fingerprinting and loads of other cool stuff.

[cwilso]

I still think we need an audio-data-level API (but much better designed, e.g. NOT in the main thread); and I also think we need hooks to enable low-level innovation like spectral frequency, pitch/time shifting, arbitrary FFT filtering, etc - but I will state that I think the number of users of those features will be far fewer (as they require much more expertise) than, say, simple panning and reverb. Not less important; just fewer.

[cristiano-belloni]

Probably they will be less, but what about the # of higher-level users indirectly using those features? Your users are developers, but they have end and intermediate users in turn. Moving audio to the web could shift its potential userbase from apps (or at least that's what anyone in this thread hopes).

[echo66]

@chrislo , one question: I know this might be a little offtopic (or not, let's see) but instead of just thinking about a DSP standard for audio, why not make it general for other devices that the browser might detect? Nowadays, the amount of sensors and input devices you have in a smartphone is bigger than any "average joe" might ever imagine. They all provide data feeds/signals to be processed in a way or another. Each device provides signals with a specific amount of dimensions: the web cam is 2D, web audio is 1D (per channel), gyroscope is 3D, compass is 8D. So, maybe W3C should start sketching a way to offer DSP operations on N-Dimensions data.

This is just a suggestion, and an offtopic as you might think. Probably, it is not even an original suggestion.

[notthetup]

I'm personally weary of pushing down more functionality into WAA spec.

I'll take ConvolverNode for example. I think the multi-threaded reverb convolver implementation is really great. It works really well in that specific use-case, but the nature of having that baked into the WAA with limited (only AudioParam) controls makes it unusable at some times.

For example, while trying to extend http://chinpen.net/auralizr/, I needed to change the impulse responses for the ConvolverNode in pseudo-realtime. The current implementations dump the convolution buffer every time the impulse response property is changed.

Now I know my use case is pretty uncommon, but since the implementation is in the spec, it can't be changed/tweaked. If the whole thing was implemented in JS (and yes I understand the issue with performance) I could have easily changed the setter for the impulse response property.

I feel there is much more to gain in working with 'userland' JS implementations of DSP functionality and getting them to perform better (asm.js, SIMD.js, etc) for the ability of tweaking them, changing them, updating them without needing to involve the browsers vendors. Faster turn around, and more control.

I understand we're not there yet in terms of JS performance, but pushing this into the spec will take time, and by the time it gets implemented, it might not have that much improved performance compared to JS implementations.

Finally, I really think this discussions needs real world numbers. I started a tiny project which tries (very unscientifically) to look at the performance of JS FFT with various libraries. Please feel free to fork/improve it. It would be great to see how much performance would a browser implementation add to this.

http://chinpen.net/webaudiofftperf/

[echo66]

@artofmus ,

var size = 2048;
var wantedSize = 1025; // I just want the first half of the spectrum.
var stdlib = {
    Math: Math,
    Float32Array: Float32Array,
    Float64Array: Float64Array
};
var heap = fourier.custom.alloc(size, 3);
// For each custom FFT, you may choose if you want Float32 or Float64. Additionally, you must say if you want to use the asm or the raw version.
var fft = fourier.custom["fft_f32_"+size+"_asm"](stdlib, null, heap);
fft.init();

var real = new Float32Array(wantedSize);
var imag = new Float32Array(wantedSize);


// Forward FFT
fourier.custom.array2heap(timeframe, new Float32Array(heap), size, size);
fourier.custom.array2heap(new Float32Array(size), new Float32Array(heap), size, size);
fft.transform();
fourier.custom.heap2array(new Float32Array(heap), real, wantedSize, 0);
fourier.custom.heap2array(new Float32Array(heap), imag, wantedSize, size);


// Inverse FFT
fourier.custom.array2heap(real, new Float32Array(heap), size, 0);
fourier.custom.array2heap(real, new Float32Array(heap), size, size);
fft.transform();
timeframe.set(new Float32Array(heap, size, size));

// Do not forget to normalize the IFFT output
for (var i=0; i<size; i++) {
  timeframe[i] /= size;
}
´´´

[cristiano-belloni]

I understand we're not there yet in terms of JS performance
by the time it gets implemented, it might not have that much improved performance compared to JS implementations

I wish, but my impression (and my fear) is that every year we're on the verge of being almost there, but we never get there. Like a "this is the year of Linux on desktop" situation.

[echo66]

Well, let's just put it this way: currently, I can't apply high quality time stretching + pitch shifting using JS implementations of FFT, without having A LOT of audio drops for more than two/three tracks playing in stereo (and I'm using a blank window!). Want a Traktor Pro in javascript? Well, too bad.

[notthetup]

@echo66 Added some info in ReadMe. I'm travelling right now as well, so don't have much time to update the UI. PM me if you need specific info.

Based on the perf test, for a window of 1k the FFT takes ~0.5msec, but the callback is every ~22msec, which seems enough time for doing multiple channels of FFT. Unless there is something wrong in the perf test (highly likely knowing me..) or your phase-vocoder implementation has other components which take a lot of time too.

[artofmus]

Сan someone say, AudioWorkerNode now available only in the form of js library (http://mohayonao.github.io/audio-worker-node/)? or in som alpha versions of any browsers that it is already available? if not when it is planned to implement?

[cwilso]

@echo66 if you're making that statement based on a JS implementation in ScriptProcessor, you're comparing apples and oranges - ScriptProcessor is both very costly and very poorly predictable due to its cross-thread nature. AudioWorker is intended to remove that problem, and the remaining stress will be from pure JS perf, which is both 1) rarely as bad as people think it is, and 2) improving daily.

@artofmus to my knowledge, no one has begun a browser implementation of AudioWorker yet.

[echo66]

@cwilso , I understand that there is a big bottleneck with ScriptProcessor but you should not forget one thing: even in a DAW like Ableton Live, there is a point when you need to "freeze" some tracks in order to avoid distortion and audio drops due to heavy CPU duties. And this happens even with C/C++ FFT implementations for effects. Of course you only face this issue if you create many tracks. But, in the browser, I wont be surprised (actually, Im betting on it) to see more audio drops and distortion than in a C/C++ app. And tools spectral analysis play have a big performance footprint: FFT is O(N log N) and even the sliding FFT is O(N). So, it is a sensitive issue.

I understand your stance regarding adding a spectral analysis node. But if you want to make the web browser very attractive to big players in music production, the issue with analysis performance must be tackle sooner or later.

I think we all just announced our stance in this issue. No need to drag this much more.

@artofmus , I advise you to take a look at that code. That polyfill/shim provides just the AudioWorker API. It does not use a different thread for AudioWorker.

[cwilso]

To be clear, I don't have a "stance", per se, on a spectral analysis node (aka a native generic FFT node) - other than I think it should be first implemented via AudioWorker and an FFT library, in Extensible Web Manifesto fashion.

You can, of course, use OfflineAudioContexts to freeze tracks. Of course, you will see more CPU load from an FFT load implemented in Javascript than one implemented in C/C++. At the same time, I think you'll find that multiplier is not as big as you think it is; modern JS engines are pretty good at compilation and type optimization, and it's completely tenable to prototype an FFT library in JS before baking that API into Web Audio.

[echo66]

@cwilso, regarding "track freezing" with OfflineAudioContext, it seems that the bug between ScriptProcessors and OfflineAudioContext still exists.

[cwilso]

Of course it does. It's not worth it to fix, as we're deprecating scriptprocessors and AudioWorkers will not have this problem.

[rtoy]

Not relevant to this issue, but what is the scriptprocessor and offline
context bug?

On Fri, May 29, 2015 at 10:49 AM, Chris Wilson [email protected]
wrote:

Of course it does. It's not worth it to fix, as we're deprecating
scriptprocessors and AudioWorkers will not have this problem.

—
Reply to this email directly or view it on GitHub
https://github.com/WebAudio/web-audio-api/issues/468#issuecomment-106885602
.

Ray

[cristiano-belloni]

@rtoy yoy can't render a scriptprocessor in an offline context. I don't have the bug report handy atm, but I can find it later.

[echo66]

https://bugs.webkit.org/show_bug.cgi?id=113274

https://code.google.com/p/chromium/issues/detail?id=374339

[cristiano-belloni]

Thanks @echo66!

[joeberkovitz]

Our plan is to support this by allowing developers to use AudioWorker for initial takes on this idea, and learn from these implementations when considering a first class spectral-domain feature in future versions of the spec.

[cwilso]

Note that the issue here is the complexity of potential options, not the lack of desire to implement such a thing. Step 1, to me, is getting AudioWorker working to remove all the huge amount of performance cost of thread-hopping in ScriptProcessor (not to mention running FFT code in the main JS thread). Step 2 is prototyping frequency domain processors using a JS FFT library, to determine what options are necessarily and sufficient. Step 2a) is prototyping moving that FFT into native code to see if the speed benefits are worthwhile. I would spitball the cost of JS FFT (in asmjs style) vs native to be MAYBE two-to-one at worst; the cost of using ScriptProcessor is far more than that in practice.

Step 3, then is deciding if there is a clear API design that emerges, and a significant benefit to the FFT being in native code.

(Also as an aside, I'd expect it would be more useful to have Math.FFT-style FFT library in native than a Web-Audio-specific one.)

[rtoy]

Marking as feature request.

[rickygraham]

Hi all. Any more on whether or not we'll see IFFT support anytime soon?

[rtoy]

If you mean the AudioFrequencyWorkerNode, then, no; it's still marked as v.next so something to consider for the next version.

If you're asking about access to an IFFT routine, that's a different question. If this is what you want, you should file another issue on that.

[rickygraham]

It is the latter. I would like access to an IFFT routine.

[mattdiamond]

This would be a pretty cool feature! If you want an example of something you can do with FFT/IFFT, I put together a frequency phase scrambler a little while back. Fun way to generate drones.

[mdjp]

This is achievable using the audio worklet (which was not available at the time the issue was raised). An FFT library is outside of the scope of the working group and should be considered as a separate spec requirement.

[Yene-Me]

This is achievable using the audio worklet (which was not available at the time the issue was raised). An FFT library is outside of the scope of the working group and should be considered as a separate spec requirement.

Hi do you have link to a documentation that explains how this can be achieved using Audio worklet. Thank you in advance