Godot: Expose real-time amplitude processing to GDscript

[MinnowMike]

The editor currently processes real-time audio amplitude (energy) and displays it in the Amplitude meter in the top right corner of the editor. It would open many sound-based game possibilities if this logic was exposed to GDScript. Unity exposes this nicely in their FMOD wrapper, it allows you to grab an array of sample amplitudes around the current play marker. From what I can tell, godot already does this in the [compute_band_energies] method of the [bands.c/h] files, but it isn't exposed anywhere to script.

---

int channel = 0;

// similar to SpatialSamplePlayer
AudioSource audio = GetComponent();

// user defined array to fill with sample amplitude values
float[] spectrum = new float[256];

// Fill the 256 array with audio sample amplitudes from the current play marker
audio.GetSpectrumData(spectrum, channel, FFTWindow.BlackmanHarris);

---

New user, but loving the structure so far, Thanks!
Michael

[ghost]

This is planned for 2.1 if I am not mistaken. The technical term for it is different but same idea. I think they call it audio routing which from what I understand it means exposing stream data as it plays which can be routed trough processes making visualizers, audio filters and much more.

See here https://trello.com/c/LVb9DWUm

Seems to be highly supported features likely to happen soon ish

[MinnowMike]

That's great news, thanks for the info! We have plans to take full advantage of it whenever it comes to fruition.

[neikeq]

We was discussing about this today on irc, and we decided to move this to 2.2

[reduz]

this is much simpler to do now with the new audio engine, but it wasn't done. Kicking to 3.1 or unless someone wants to do it.

[SL-RU]

It will be very useful feature in many games.

[Calinou]

@SL-RU Please refrain from bumping issues without significant new information; use the :+1: reaction button on the first post instead.

[LikeLakers2]

I'm currently working on a audio-reactive layout for videos using Godot. This would be extremely useful and would let me save some time instead of having to record the data from Unity or get the data through some manual means. I hope this gets in for 3.1, if not earlier than that.

[MinnowMike]

I learned that the AudioStreamPlayer seems to expose the raw audio data in stream.data as a PoolByteArray, and started thinking this might be possible now. I wrote a working mock-up of the idea in RayLib, then tried to reproduce it in GDScript but got stuck on trying convert the raw audio PoolByteArray to usable samples (PoolRealArray? but that doesn't seem to be exposed to GDScript). StreamPeerBuffer is supposed to provide array conversion, but I don't understand how to use it. I can calculate a sufficiently accurate play head position using the sample rate and a calculated elapsed time to give me a position in the array to grab samples, but don't know what the bytes need to be converted to or how to convert. If the PoolByteArray data matches the structure documented in the binary serialization API, then really all that would be needed is a conversion from bytes to float. I could offset my play head to the closest index of the beginning of any given byte array comprising a float.

[Bauxitedev]

@MinnowMike I've managed to do this in my spectral visualizer by using C# and some methods to convert raw byte arrays into shorts and floats. You can look at the source code to see how it works. It does frequency analysis as well, but to get amplitude information you could just take a window of e.g. 512 samples and calculate the RMS over it.

As mentioned, this is not ideal though, because it needs manual tracking of the playhead position, in addition to keeping the entire sample in memory.

[akien-mga]

Moving to the next milestone as 3.1 is now feature frozen.

[KoBeWi]

Isn't it this? https://godotengine.org/article/godot-32-will-get-new-audio-features

[SL-RU]

Isn't it this? https://godotengine.org/article/godot-32-will-get-new-audio-features

Yes, this is it!

[KoBeWi]

So this can be closed.