Just_audio: Rewrite iOS implementation based on AVAudioEngine

[ryanheise]

Is your feature request related to a problem? Please describe.

Certain features such as a visualizer (#97 ), equalizer (#147 ), pitch shifting (#329 ) may be more easily implemented if based on AVAudioEngine rather than the current AVQueuePlayer.

Describe the solution you'd like

Either reimplement using AVAudioEngine directly, or use an AVAudioEngine-based library such as AudioKit.

Describe alternatives you've considered

We can get some of the way there by plugging an audio tap processor into AVQueuePlayer, but unfortunately this does not give us access to iOS's built-in pitch shifting API which is only available via AVAudioEngine. This could still be possible via the audio tap processor by manually implementing the pitch shifting algorithm, or perhaps integrating the C version of sonic, but long term an AVAudioEngine-based implementation may end up being more flexible in terms of implementing other audio processing features.

Additional context

None

UPDATE

This will be a collaborative effort. Anyone can contribute, and we will be following the plan in this shared Google Doc:

https://docs.google.com/document/d/17EZEvmiyn94GCwddBGS5BAaYer5BTRFv-ENIAPG-WG4/edit?usp=sharing

We are currently in the research phase, so you can contribute by sharing any relevant links to useful resources you have found to help implement each feature in the list.

[mvolpato]

Is it an option to move to Swift?

[ryanheise]

Yes, this probably will be written in Swift.

[nt4f04uNd]

this will be breaking for folks, don't do this

[ryanheise]

@nt4f04uNd Do you mean breaking in terms of stability, or breaking in terms of codec support?

[nt4f04uNd]

in terms of that you can have these configs, except one

| app | library |
| obj-c | obj-c |
| swift | obj-c |
| swift | swift |

library written with swift can't be used with obj-c projects

[ryanheise]

That's true, although just to play devil's advocate:

• New Flutter projects have been Swift by default now for more than a year.
• People are converting older Objective C projects to Swift.
• Consequently, this is a vanishing problem, and people who have it can solve it with flutter create.

This will also virtually only affect the more experienced Flutter developers (who created their project before Flutter changed the default template to Swift), so if we include instructions in the README on how to convert their project from Objective C to Swift, I wouldn't expect Swift to be a showstopper.

It would be interesting to know the actual statistics on how many people are still running Objective C projects. Plugins are a different story, as people may have various reasons to choose Objective C vs Swift when writing their plugin in that language. But a project doesn't actually contain any Objective C or Swift code so it really doesn't make a difference whether you switch a project from Objective C to Swift except that it just opens the door to accessing all of the Swift plugins on pub.dev.

There are some instructions in the README for Android on how to convert old projects to the latest V2 plugin architecture, and in a similar vein I could add instructions for how to update an old Objective C project to Swift.

[ryanheise]

(Copying this comment from another issue to get broader interest)

Is this supported on iOS currently?

The waveform visualizer is implemented on iOS but not pitch. You can track the pitch feature here: #329

There is a big question at this point whether to continue with the current AVQueuePlayer-based implementation or switch to an AVAudioEngine-based implementation. For pitch scaling, I really want to take advantage of AVAudioEngine's built-in features, but that requires a rewrite of the iOS side - see #334 and this is a MUCH bigger project.

I would really like to see an AVAudioEngine-based solution see the light of day, but it will probably not happen if I work on it alone. If anyone would like to help, maybe we can pull it off with some solid open source teamwork. One of the attractive solutions is to use AudioKit which is a library built on top of AVAudioEngine which also provides access to pitch adjustment AND provides a ready-made API for a visualizer and equalizer. That is, it provides us with everything we need - BUT it is written in Swift and so that involves a language change and it means we may need to deal with complaints that old projects don't compile (we'd need to provide extra instructions on how to update their projects to be Swift-compatible).

Would anyone like to help me with this? (Please reply on #334)

[ryanheise]

Another interesting library: https://github.com/tanhakabir/SwiftAudioPlayer

This suggests we want a combination of AVAudioEngine and AudioToolbox:

Thus, using AudioToolbox, we are able to stream audio and convert the downloaded data into usable data for the AVAudioEngine to play. For an overview of our solution check out our blog post.

Given that I'm spread a bit thinly, I would not like to make this a solo effort, so I will wait until some more people post below who might be willing to team up and work together.

One other thought I had is that I may want to move the current iOS implementation out into its own federated plugin implementation rather than bundling it with the main plugin, although it can still be endorsed so the dependency automatically gets added to your app. The advantage of this is that if we can eventually create an AVAudioEngine-based implementation of the just_audio API, we can now do this without throwing away the old implementation. In case there are some features that don't work in the new implementation, users will still be able to use the old implementation, and vice versa.

[nt4f04uNd]

blacklist the https://github.com/tanhakabir/SwiftAudioPlayer. i tried to use it and it's horrible, the audio glitches and author barely maintains it
if you are researching variants of libraries you could use, i saw https://github.com/tumtumtum/StreamingKit (it's FOSS by the way)

imho you should just stick to the most popular one, i.e. AudioKit

[nt4f04uNd]

i like the idea of federated implemenation, in fact, this should be done for each platform

[ryanheise]

blacklist the https://github.com/tanhakabir/SwiftAudioPlayer. i tried to use it and it's horrible, the audio glitches and author barely maintains it
if you are researching variants of libraries you could use, i saw https://github.com/tumtumtum/StreamingKit (it's FOSS by the way)

imho you should just stick to the most popular one, i.e. AudioKit

I agree with you although what I did find interesting about SwiftAudioPlayer is _not_ that I want to use it as a library, but rather that the author has written an informative blog post explaining the additional components that were needed on top of AVAudioEngine, which may also apply to us even if we go with AudioKit. For example, to stream audio, we will probably need to use techniques similar to those used in SwiftAudioPlayer to first "get" the audio to then feed into AudioKit.

Thanks for sharing StreamingKit, the name itself implies that it will be another good reference when trying to implement the streaming part of this.

[mvolpato]

Hi @ryanheise, I would be happy to help but I do have some doubts:

1. I have no experience with development of audio components.
2. I have no experience with developing a plugin for Flutter.

I do have experience with Swift, and that would be my preference over Objective-C.

So I would be more comfortable with a list of clear (smaller) tasks to follow, and not just "Implement AudioKit" or similar. I do not know if this is what you are looking for.

[nt4f04uNd]

fyi StreamingKit is not just for streaming, it to my understanding offers pretty much the same set of features as AudioKit

[ryanheise]

@mvolpato I'm happy to have your support! We definitely need a plan which can be broken down into tasks that can be each done by different people.

What I'll do is create a Google Doc with a list of features that need to be supported, and the first thing that needs to be done is to just collecting links to relevant documentation and tutorials or StackOverflow answers relevant to implementing that feature. That first research phase will give us the peaces of the puzzle we need to then prioritise the tasks and start implementing them.

So the plan is:

1. Create a list of features to implement
2. Collect research for each feature
3. Work out what needs to be done first
4. Start implementing

The 3rd point is not in strict order so we can start thinking about that earlier, but I think the order in which to do things will fall out naturally. e.g. First we should implement loading a simple audio source, then playing, then pausing, then the other standard controls. State broadcasting could happen in parallel with this.

I'll also need to move the current iOS implementation into a separate platform implementation package.

I'll post a link to the doc once I create it.

[ryanheise]

Shared doc: https://docs.google.com/document/d/17EZEvmiyn94GCwddBGS5BAaYer5BTRFv-ENIAPG-WG4/edit?usp=sharing

I will update the top post with details.

[ryanheise]

Hi @mvolpato I'm ready to get ball rolling using a Swift-based implementation. Swift is nice, but there seem to be complications in how the compiler works.

Would you be able to try these steps out below in your environment and see if they work for you?

First, create a new plugin from the template:

flutter create -t plugin --platforms android,ios audiokit

Then in ios/SwiftAudiokitPlugin.swift I import AudioKit and make a symbolic reference:

import AudioKit
...
var mixer: AKMixer

Then in ios/audiokit.podspec I added the cocoapods dep. Tried various alternatives, none of which worked, though:

  s.dependency 'AudioKit/Core', '~> 4.0'
  # s.dependency 'AudioKit/Core', '4.11'
  # s.dependency 'AudioKit/Core', :git=>'https://github.com/AudioKit/AudioKit/', :branch => 'v5-develop'

If you try to run the example directory, you get the error mentioned here: https://github.com/AudioKit/AudioKit/issues/2267

Would be interested if you could try the above steps also and see if it works for you. Note that I also upgraded my cocoapods and Xcode to the latest versions before running this.

[ryanheise]

If it also fails for you (and a solution doesn't start out) then we may need to create either a Flutter issue or an AudioKit issue. According to reports in the above AudioKit issue, the error was resolved, so maybe its an issue that occurs due to the Flutter setup specifically.

[mvolpato]

'AudioKit/Core', :git=>'https://github.com/AudioKit/AudioKit/', :branch => 'v5-develop'

It looks like cocoapods is not supported (yet) for version 5, so this will not work for sure.

I also cannot get it to work. I tried different versions.

[mvolpato]

It looks like this other plugin has it working. I did not have time to investigate their approach yet. I will check later today.

[ryanheise]

Great find! I think what's happening on my project (and my environment) is that it resolving AudioKit to 4.11 whereas flutter_sequencer is resolving it to 4.11.1, and according to that issue, the bug fix for it requires 4.11.1 or later.

I was scratching my head for a while but it was caching the first version it ever resolved to when I made my first attempt at the podspec, even after running flutter clean. Turns out flutter clean does not delete the Pods directory and also doesn't delete the Podfile.lock file. This is why flutter_sequencer ends up resolving the AudioKit dependency to 4.11.1 when there's an even later version 4.11.2 available. Anyway, deleted everything and with a fresh start it works!

I think next it will probably be necessary to move the current iOS implementation into a separate package like the web one, then the new AudioKit implementation can just be another alternative to that.

[mvolpato]

Ok, I will start working on this, but

I think next it will probably be necessary to move the current iOS implementation into a separate package like the web one, then the new AudioKit implementation can just be another alternative to that.

I do not get it why this is needed? Would that mean that users will be able to choose between the current implementation with just_audio and the AudioKit one with just_audio_ios_audiokit?

[mvolpato]

@ryanheise in this branch I just copied the iOS implementation in a separated folder. Let me know if this is what you meant. I will remove Android code from it.

[ryanheise]

Hi @mvolpato I actually haven't fully experimented with the different approaches, but the goal is to use the federated plugin architecture to allow people to opt in to the AudioKit implementation. This is because in the beginning we can expect it to lack features and stability compared to the original implementation, even if it may also implement some new features, so people will then be able to choose the federated implementation they prefer.

But maybe the mechanism we need to do this is possible even without moving the iOS implementation into a separate package (I have not tested that). It could be worth trying that first and if it fails then move the iOS implementation into a separate package. In the latter case, it should also be endorsed in the main just_audio pubspec.yaml file in the same way the web implementation is endorsed.

Also I will probably create a new branch just for AudioKit development, and that in turn will branch off dev. I will try to set this up some time today.

[mvolpato]

If we separate the AudioKit based plugin from the rest (like the web one) then branching is not really necessary.

I will start with a new plugin, following the setup of the web one. Then I will either PR against the new branch or the old dev.

[nt4f04uNd]

the doc is public (i don't know if this is intended)
i added yet another engine solution there https://github.com/sbooth/SFBAudioEngine/

[ryanheise]

It's intended :-) Thanks for adding another resource!

[ryanheise]

I think next it will probably be necessary to move the current iOS implementation into a separate package like the web one, then the new AudioKit implementation can just be another alternative to that.

While this is theoretically possible, we do lose the ability to symlink the iOS and macOS implementations because pub has a security restriction that prevents symlinks outside the project root, and we'd now want to share code across the separate just_audio_macos and just_audio_ios projects.

Maybe instead of having a separate package for the ios and macos platforms, we should have just one, just_audio_darwin, which includes both the iOS and Android platform implementations within the same project. Then we could have just_audio_audiokit_darwin for the alternative implementation, again containing both the ios and macos implementations.

[mvolpato]

As a user of the package, how would you select in the pubspec which ios version to use? Does it matter if you want to use the current Android implementation and the new AudioKit for iOS?

[ryanheise]

I'm looking into this now, but it seems the federated architecture doesn't fully support what we ant to do anyway.

Although the federated plugin design doc indicates that there is such a thing as a "default" implementation for a platform, implying that it can be overridden, it seems that isn't actually supported. If there is no default, an app can just add just_audio_audiokit to its dependencies and it will work. But if we try to provide a default, both end up getting added and things are non-deterministic.

I have created an issue to request at least this problem to be fixed: https://github.com/flutter/flutter/issues/80374

But as to your question of Android, the Android implementation would not be in the same package as the iOS/macOS package, so you would theoretically be able to vary these independently (subject to the problem of overriding defaults being resolved)

[ryanheise]

There is a solution being worked on by the flutter team which will probably allow the default implementation to stay within the main just_audio plugin. So it should be fine to create a new directory alongside just_audio_web called just_audio_audiokit. It won't work until that solution is ready, but to workaround that in the short term we can just temporarily comment out the default iOS implementation's registration code:

+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar>*)registrar {
    /*
    FlutterMethodChannel* channel = [FlutterMethodChannel
        methodChannelWithName:@"com.ryanheise.just_audio.methods"
              binaryMessenger:[registrar messenger]];
    JustAudioPlugin* instance = [[JustAudioPlugin alloc] initWithRegistrar:registrar];
    [registrar addMethodCallDelegate:instance channel:channel];
    */
}

The audiokit plugin also needs to have a different class name from the default plugin because the flutter tool will still try to add both to the generated plugin registrant and they will conflict if two plugin classes have the same name.

So maybe we can call the built-in one JustAudioPlugin and the AudioKit one JustAudioAudioKitPlugin.

[mvolpato]

So maybe we can call the built-in one JustAudioPlugin and the AudioKit one JustAudioAudioKitPlugin.

JustAudioKitPlugin :)

Ok, i will look into it in the coming days