Just_audio: Provide a simple Audio Visualizer

[oseiasmribeiro]

Provide a simple Audio Visualizer (showing low, medium and high frequencies) that offers the possibility of increasing or decreasing a number of bars. This is useful for the user to make sure that the audio is being played or recorded in the application. Sometimes the volume or mic can be at a minimum.

Example: https://dev.to/armen101/audiorecordview-3jn5

[ryanheise]

I also think this feature would be a good idea, although it is not necessarily the highest on my personal priority list. I will of course welcome pull requests.

The underlying API on the Android side to implement this is:

https://developer.android.com/reference/android/media/audiofx/Visualizer

This API is relatively straightforward to use.

However, on iOS, it does not appear to be as straightforward.

[oseiasmribeiro]

Thanks for your answer! I look forward to and await this resource.

[sachaarbonel]

• Someone made a plugin (only Android) FlutterVisualizers
  -And there is this iOS implementation : DisPlayers-Audio-Visualizers written in objective c

[oseiasmribeiro]

@sachaarbonel thanks!

[esDotDev]

Any chance you might add this? It's not present in any sound libraries for Flutter currently, and would be really nice to be able to build visualizers.

Edit, looking into it a bit further, I think we just need this value on iOS, is that a big lift?
https://developer.apple.com/documentation/avfoundation/avaudioplayer/1390838-averagepowerforchannel?language=objc

[ryanheise]

That would allow for a rudimentary visualiser, although probably what we want is something equivalent to Android's API, so we'd want to do a FFT on the audio signal.

After accidentally stumbling upon it, it seems there is a way to do this. First, we create an AVMutableAudioMix and set it in AVPlayerItem.audioMix. To this audio mix's inputParameters array we add an instance of AVMutableAudioMixInputParameters. And on this instance we can access the audioTapProcessor through which it should be possible to analyse the audio signal and do the FFT.

[pstromberg98]

@ryanheise I think the best solution for audio visualization is providing a clean way to subscribe or retrieve the samples/buffers of audio data. Providing a way to pull in the raw PCM data would allow for more than just FFT analysis but would open the door to nearly any other type of audio analysis.

The audioTapProcessor could be the way to access the raw data on iOS and then making use of the Renderer in exoplayer to access the raw data for android. I imagine the api would be as simple as a stream on the audio player called rawDataStream or sampleStream or something akin to that. What are your thoughts on that?

[ryanheise]

@pstromberg98 that's possibly a good idea. Looking at the Android Visualizer API, it actually provides both the FFT data and the waveform data, so we could do the same.

Although you could argue we only need the latter, both Android and iOS provide us accelerated implementations of FFT which we should take advantage of.

[pstromberg98]

@ryanheise Totally! It would be super nice for the library to provide the fft and I don’t see any harm in providing that. I was mainly saying in the case of having either one or the other it would probably be better to provide the raw waveform just in case users wanted to do other analysis and transforms on the data. But having both would be slick!

[ryanheise]

I've just implemented the Android side on the visualizer branch. Call:

samplingRate = player.startVisualizer(
    enableWaveform: true,
    enableFft: true,
    captureRate: 10000,
    captureSize: 1024);

Then listen to data on visualizerWaveformStream and visualizerFftStream. The returned sampling rate can be used to interpret the FFT data. Stop the capturing with player.stopVisualizer().

The waveform data is in 8 bit unsigned PCM (i.e. subtract 128 from each byte to get it zero-centred). The FFT is in the Android format, and I'm not sure yet whether the iOS native format will be different, so that particular part of the API may be subject to change.

It may take a bit longer for me to get the iOS side working, but in the meantime would anyone consider contributing a pull request that adds a very minimalistic visualiser widget to the example, demonstrating how to make use of the data in visualizerWaveformStream?

[ryanheise]

Information on how to interpret the Android FFT data:

https://developer.android.com/reference/android/media/audiofx/Visualizer#getFft(byte[])
https://stackoverflow.com/questions/4720512/android-2-3-visualizer-trouble-understanding-getfft

As for iOS, some ideas for implementation:

https://stackoverflow.com/questions/22751685/using-mtaudioprocessingtap-for-fft-on-ios
https://chritto.wordpress.com/2013/01/07/processing-avplayers-audio-with-mtaudioprocessingtap/ (not exactly relevant)

Some official Apple stuff:

https://developer.apple.com/documentation/avfoundation/avplayeritem/1388037-audiomix?language=objc
https://developer.apple.com/documentation/avfoundation/avaudiomix/1388791-inputparameters?language=objc
https://developer.apple.com/documentation/avfoundation/avmutableaudiomixinputparameters?language=objc
https://developer.apple.com/documentation/avfoundation/avmutableaudiomixinputparameters/1389296-audiotapprocessor?language=objc
https://developer.apple.com/forums/thread/11294 (forum post)
https://codedump.io/share/KtEfM7VG0wrL/1/how-do-you-release-an-mtaudioprocessingtap (releasing tap)

[ryanheise]

I've changed the API slightly to include the sampling rate in the captured data, and included a very simple visualiser widget in the example. The iOS side will be more difficult and I have some higher priority things to switch to for the moment but help is welcome. In particular, if you would like to either give feedback on the API, contribute a better visualiser widget example or even help getting started on the iOS implementation using the documentation linked above.

[ryanheise]

To those interested in this feature, would you like a separate method to start the request permission flow to record audio for the visualizer? Currently startVisualizer() will start this flow when it detects that permission hasn't already been granted, but perhaps some apps would like more control over when permission is to be requested.

[pstromberg98]

@ryanheise Personally I like when libraries provide more fine grain control but I can see the argument for both. Perhaps it would be best to have startVisualizer() request permissions (if needed) by default but also have a way to request the permission separately.

[defsub]

I second the comment from @pstromberg98

[ryanheise]

I've made an initial implementation of the visualizer for iOS. Note that this is definitely not production ready. Some problems to investigate:

1. Against Apple recommendations, I allocate an NSData buffer for each capture inside the TAP. You might want to keep an eye on memory usage in case this causes a leak.
2. I've only crudely converted the samples into 8 bit unsigned PCM. To make it look closer to Android, I fudged the scaling by 3x'ing every sample. Not really sure what Android is going under the hood so I can't emulate it exactly.
3. There's a chance this might not work with different audio formats, and I may need to add special cases for the different formats.
4. FFT is not implemented yet, only the raw waveform data.

Thanks, @pstromberg98 for the suggestion. I agree, and I'll try to implement that.

As before, I unfortunately need to work on some other issues for a while, particularly null safety. But hopefully this commit provides a good starting foundation to build on.

Contributions are also welcome, so here is the "help wanted":

• FFT code
• A better visualizer widget
• Testing on iOS for memory leaks
• Testing on iOS for support of various audio formats

[ryanheise]

Has anyone been able to give this a whirl yet? I think this would be a really useful feature to include in the next release, so I'd like to make it a priority, although for that to happen, it would definitely help to get some feedback on the iOS side in terms of memory efficiency and stability. I will of course eventually add the option to start the permission flow on Android on demand, but I think the iOS stability will be the most critical thing to be confident about before I include this in a release, along with the iOS FFT implementation.

Of course, I could just document it as being experimental and unstable, and release it that way, which might actually not be a bad idea to get more eyes on it.

[pstromberg98]

@ryanheise If I can find time I will talk a look at the iOS side and give my thoughts and feedback. I appreciate your efforts on it so far and am eager to jump in and help when I find time 👍.

I think marking the feature as experimental would make a lot of sense.

[MichealReed]

I pulled it over and merged the changes to check out the android version, but cant seem to get it working.

Mic permission is granted (although it crashes app after prompt acceptance), but it prevents my player from playing anything. I even tried wrapping it in a future to ensure the player has data before calling. Any ideas?

Future.delayed(Duration(seconds: 2), () {
        var samplingRate = activeState.player.startVisualizer(
            enableWaveform: true,
            enableFft: true,
            captureRate: 48000,
            captureSize: 1024);
        activeState.player.visualizerWaveformStream.listen((event) {
          print(event);
          this.add(AudioManagerVizUpdateEvent(vizData: event.data));
        });
      });

[ryanheise]

Thanks for testing that. It turns out there is another change to the behaviour of ExoPlayer in that onAudioSessionIdChanged is not called initially for the first value. I've done the merge and fixed this issue in the latest commit.

[ryanheise]

Perhaps. With the null safety release of Flutter soon to reach stable, I'm not sure if I'd like to do this before or after that. Currently I'm maintaining two branches which is a bit inconvenient to keep in sync.

We'll see how things pan out but first I may need to focus on getting the null safety releases ready.

[MichealReed]

Thanks for jumping to this. Things are working great so far, but the fft buffer visual is a bit different than I expected from using my custom visualizer on other fft sources. Will try to take a deeper look and report back the bug if I find it.

[ryanheise]

That's basically just the raw output from the Android API which is documented here:

https://developer.android.com/reference/android/media/audiofx/Visualizer#getFft(byte[])

So it's possible you might get weird output unless you interpret that byte array as per the above documentation.

[penalosa]

Is this supported on iOS currently? I tried using the visualiser branch, but I keep getting an error: flutter: setPitch not supported on this platform

[ryanheise]

Is this supported on iOS currently?

The waveform visualizer is implemented on iOS but not pitch. You can track the pitch feature here: #329

There is a big question at this point whether to continue with the current AVQueuePlayer-based implementation or switch to an AVAudioEngine-based implementation. For pitch scaling, I really want to take advantage of AVAudioEngine's built-in features, but that requires a rewrite of the iOS side - see #334 and this is a MUCH bigger project.

I would really like to see an AVAudioEngine-based solution see the light of day, but it will probably not happen if I work on it alone. If anyone would like to help, maybe we can pull it off with some solid open source teamwork. One of the attractive solutions is to use AudioKit which is a library built on top of AVAudioEngine which also provides access to pitch adjustment AND provides a ready-made API for a visualizer and equalizer. That is, it provides us with everything we need - BUT it is written in Swift and so that involves a language change and it means we may need to deal with complaints that old projects don't compile (we'd need to provide extra instructions on how to update their projects to be Swift-compatible).

Would anyone like to help me with this? (Please reply on #334)

• I will copy this comment to related issues.