Audiokit: crash format. SampleRate == hwFormat. SampleRate

[cs571393]

When I start AKMicrophone() initializing, required condition is false: format. SampleRate == hwFormat. SampleRate leads to crash.
I want to ask whether I need to modify the AudioKit. Engine. InputNode. InputFormat (forBus: 0). SampleRate. Or how to solve the problem

    AKSettings.sampleRate = 44100
    AKSettings.bufferLength = .veryLong
    do {
        if #available(iOS 10.0, *) {
            try AKSettings.setSession(category: .playAndRecord, with: [.allowBluetoothA2DP, .allowBluetooth, .defaultToSpeaker])
        } else {
            try AKSettings.setSession(category: .playAndRecord, with: [.allowBluetooth, .defaultToSpeaker])
        }
    } catch {
        AKLog("Could not set session category.")
    }

    AKSettings.defaultToSpeaker = true
    AKSettings.useBluetooth = true
    AKSettings.playbackWhileMuted = true
    AKSettings.audioInputEnabled = true

    self.mic = AKMicrophone()

[cs571393]

AudioKit.engine.inputNode.inputFormat(forBus: 0).sampleRate
value is 16000

[roussetjc]

@cs571393 Do you use any external devices with a 16K sample rate?
I am struggling with the microphone sample rate crash for quite a long time now too. My app connects to different USB microphones with different sample rates (48K, 96K or 192K) and I still can't make this work seamlessly when plugging/unplugging.
There seems to be something wrong with AudioKit.deviceSampleRate which is not responding to any kind of sample rate setting/formatting.

[cs571393]

@roussetjc Yes，My devices is 16K sample rate.

[roussetjc]

Does it work with AKSettings.sampleRate = 16000?

[warpling]

I often see the same crash when switching to AirPods

[roussetjc]

When an AKMicrophone is initialized, the function @objc public init?(with format: AVAudioFormat? = nil) is called.
Let's say we pass a format with a 16K sample rate. Everything is fine until...

#if os(iOS)
// we have to connect the input at the original device sample rate, because once AVAudioEngine is initialized, it reports the wrong rate
do {
    try setAVSessionSampleRate(sampleRate: AudioKit.deviceSampleRate)
} catch {
    AKLog(error)
    return nil
}

Most of the time, AudioKit.deviceSampleRate is equal to 48K. The init then crashes at AudioKit.engine.connect(AudioKit.engine.inputNode, to: self.avAudioNode, format: format ?? formatForDevice). I don't really understand where this AudioKit.deviceSampleRate comes from nor how to force it to the desired value (a device can work with several compatible sample rates).

[ClassicalDude]

I have been struggling with this issue for a long time. Older devices use a sample rate of 44.1K, so I have to implement my code twice - for 44.1 and for 48. There doesn't seem to be a way right now to force a sample rate on the microphone. You have to set the sample rate of the hardware.

Please note that if you're using bluetooth devices, your sample rate will indeed be in the area of 8K to 16K - the connection is not fast enough to get beyond that, even with a2dp.

[warpling]

So the issue arrises when AudioKit and the AKMicrophone are set to different sample rates and then connected together? So is the solution something along the lines of retrieving the sample rate from the current device AKMicrophone is using and setting the AVSessionSampleRate to that first?

[ClassicalDude]

This issue has been discussed before, but I'll try to lay out the solution that I have working in a stable manner for me. This took a lot of trial and error, and the order of things makes a big difference.
Start by defining the AudioKit variables and constants you are using. In my case, I am doing it outside of any class, on the app level, because I need to access them from several classes.

// AudioKit global variables
let mic = AKMicrophone()
var boost = AKBooster()

Then, inside my viewController, I set the sample rate to match the hardware:
AKSettings.sampleRate = AudioKit.engine.inputNode.inputFormat(forBus: 0).sampleRate

Then I set the input device:

if let inputs = AudioKit.inputDevices {
    do {
        try AudioKit.setInputDevice(inputs[0]) // Or other existing input, I query them by name
    } catch {
        print ("Could not set audio inputs: \(error)")
    }
    do {
        if mic != nil {
            try mic?.setDevice(inputs[0])
            print ("Microphone set to \(inputs[0])")
        } else {
            print ("mic was nil")
        }
    } catch {
        print ("Could not set the audio input device to the AKMic: \(error)")
    }
}

Next - connect some nodes, start the engine and then set the session:

if !AudioKit.engine.isRunning {
    do {
        // I set up a dummy output, because I don't need one. But there must be an output.
        AudioKit.output = AKBooster()
        boost = AKBooster(mic) // Another booster, this one from the mic 
        AKSettings.audioInputEnabled = true
        try AudioKit.start()
        // Next line - if you need bluetooth support
        try AKSettings.setSession(category: .playAndRecord, with: .allowBluetoothA2DP)
        // Next line - buffer size, if you need to change it         
        try AKSettings.session.setPreferredIOBufferDuration(0.02)
        print ("Audio sample rate is: \(AVAudioSession.sharedInstance().sampleRate)")
        print ("Audio buffer duration is: \(AVAudioSession.sharedInstance().ioBufferDuration)")
    } catch {
        print ("Could not start AudioKit: \(error)")
    }
}

Next, connect any other nodes you need to the mic's booster, then set start the mic, then start the booster, then set the booster's gain. If you need to set the type of AVSession you're using, this comes after that:

do {
    try AKSettings.session.setMode(.measurement) // Or any other mode
    print ("Audio mode is: \(AVAudioSession.sharedInstance().mode)")
} catch {
    print ("Could not set Audio Session Mode: \(error)")
}

Finally, remember that if the sample rate makes a difference to your purposes (it does to mine, since I am doing fft analysis and the results are very different), you have to get the actual sample rate used and implement your code accordingly:

switch AKSettings.sampleRate {
case 48000.0:
        // code
case 44100.0:
        // code
default:
        // code
}

If you need to suspend any of the audio chain, just stop the nodes - don't stop the engine. And if there are any performance issues (and these may vary between different generations of the processors), it is a good idea direct the code to the main thread by wrapping it with DispatchQueue.main.async.

I hope this helps.

[warpling]

@ClassicalDude thanks for writing all this up! Both for me but others who come along.

In my app I don't always need the microphone. When AKMicrophone is initialized it, through a side-effect, causes the haptic engine to be shut down until the engine it is associated with is deinit'd. For this reason I only initialize AKMicrophone when I need to move the app into a recording state. When I do so I first stop the engine, then configure the mic, attach it, and start the engine back up. This seems to result in crashes when a bluetooth headset is connected.

I've created a test project to reproduce this crash. It uses a fairly basic setup and the following to cause a crash.

Start the engine without the microphone
Stop the engine
Start the engine with the microphone

I'd love to fix this problem with in AudioKit but could use some guidance on how the sample rates work. When you call AKSettings.sampleRate = AudioKit.engine.inputNode.inputFormat(forBus: 0).sampleRate is this sample rate the sample rate of the currently selected AKDevice for the initialized AKMicrophone?

[ClassicalDude]

I am afraid I am not that much of an expert. Getting to a dependable, stable state of my project took a lot of experimentation, and early on I discovered that stopping the audio engine is not a good idea, and that the order of your actions matters a great deal.
You can keep trying different combinations, and hopefully you'll find one that works. As far as my understanding goes, and it can be wrong, the AKSettings.sampleRate line gets the sample rate from the built-in audio card. The moment you attach a bluetooth device, the system's sample rate automatically changes to that of the device, and if you try to start the audio engine with the old sample rate, you'll get a crash. I managed to get my app to output to a bluetooth headset while taking input from the audio card, and you have my code for that above - but I never stop the engine.
Just a thought - try to set a notification for a change of audio routes:

NotificationCenter.default.addObserver(
            self,
            selector: #selector(audioRouteChanged(notification:)),
            name: AVAudioSession.routeChangeNotification,
            object: nil)

Use the called function to query the system's current sample rate, and try to set the sample rate again, before reattaching the mic and restarting the engine. No guarantees.

[warpling]

Linking in #1659 which is definitely related and contains a good wealth of info on this too.

[warpling]

@ClassicalDude It took me awhile to find a stable setup as well. Hence how I got to this crazy setup where I stop and start the engine depending on microphone use.

The major discovery that I made today is that AKMicrophone can be initialized and attached later just fine so long as AudioKit.engine.inputNode is touched before the first engine start() call (this could be as simple as printing the inputNode to trigger it's lazy initialization). This kills haptics unless the iOS 13 flag AKSettings.allowHapticsAndSystemSoundsDuringRecording is set to true but the resulting microphone automatically switches sources and sample rates on its own just fine.

Until I can go through AK's code more with @aure I'll probably fix my own setup by just initializing the microphone at startup (rather than trying to later to avoid touching the inputNode).

[eljeff]

For what its worth - I just solved an issue with the microphone by reading AVAudioSession.sharedInstance().sampleRate early on and setting AKSettings.sampleRate to that.

I also made that public now in audiokit so you will soon be able to do AudioKit.deviceSampleRate

[warpling]

@eljeff could you provide a bit more context. You save off AVAudioSession.sharedInstance().sampleRate early on but when do you use that to rest AKSettings.sampleRate? Before instantiating AKMicrophone?

[mglodack]

@warpling I was experiencing this exact same problem specifically when the input/output route changed on my iPhone. I do the following:

1. Subscribe and handle the route change notification
2. Whenever I receive a route change notification and I'm in a recording mode (meaning I have an instance of AKMicrophone attached the AVAudioEngine) I tear down the microphone and other AudioKit nodes, transition to a readyToRecord mode, and wait for the user to take action.
3. Before I configure the AKMicrophone, the other AudioKit nodes, and invoke the AudioKit.start() I set the following:

AKSettings.sampleRate = AVAudioSession.sharedInstance().sampleRate

I tried a handful of route change test scenarios and so far I have yet to experience this crash. 🤞

I hope this helps 😄

[warpling]

@mglodack thanks for the steps! I might move back to a more dynamic setup like this where AKMicrophone is only init'd when needed since I can't things to behave well when the mic is persistent.

Just to clarify, you're saying at app startup you do not necessarily init a mic and instead wait until the mic is needed?

[mglodack]

Just to clarify, you're saying at app startup you do not necessarily init a mic and instead wait until the mic is needed?

Yes, I wait until the mic is needed and tear it down when it's no longer needed.

[romancfischer]

I can't confirm that initializing the microphone later on would work. Tried printing inputNode early on but it doesn't work in my case.

I made the discovery yesterday that Audiobus doesn't launch my app as long as AKMicrophone is initiated. There seems to be a bug which is eventually corellated to this issue here.

[warpling]

@mglodack Doesn't take take 1s or so every time you start/stop the engine? That used to be the case for me so I've been trying to avoid it if I can instead just disable the mic and "hot swap" it in or activate it when needed. Initing the mic at start-up is causing all background audio to be paused which is very unideal but at least crashes seem to be mitigated.

@romancfischer I used to init the AKMicrophone later but around 10% of users experienced a crash because of this. Some of these fixes may help but there is definitely something wrong at the moment that we need to get to the root of.

[mglodack]

@warpling I believe when I did my initial profiling using my iPhone X on both iOS 12 and iOS 13 the most expensive AudioKit call was AudioKit.start() which took on average ~300-400ms. While that is noticeable to the human eye, I did some smoke in mirrors to animate a button for that same duration and it's not even noticeable.

Also, I recently revamped my use of AudioKit to be on a separate DispatcheQueue. This has made my own interactions with AudioKit more complex from a developer perspective, but it guarantees all work is done serially and no work is done on the main thread.

[eofs]

Hi all,

Here's how I was able to reproduce the issue reliably on iOS:

1. Launch the app
2. Play an audio file using AVPlayer. The audio file I used was sampled in 44.1kHz.
3. Call AKSettings.setSession(category: .playAndRecord)
4. Call AKMicrophone(with: nil)
5. Crash: required condition is false: format.sampleRate == hwFormat.sampleRate

The format had sample rate of 48kHz while hwFormat had sample rate of 44.1kHz.

(lldb) po AVAudioSession.sharedInstance().sampleRate
44100.0

(lldb) po AudioKit.deviceSampleRate
48000.0

I was able to fix the issue by re-implementing AKMicrophone. The code is basically a copy-paste from the original class but I removed the calls to setAVSessionSampleRate() method.

Also, I modified the AKMicrophone.getFormatForDevice() method to avoid using AudioKit.deviceSampleRate and use AVAudioSession.sharedInstance().sampleRate instead.

Here are the relevant changes in code:

@objc open class AKMicrophoneWorkaround: AKNode, AKToggleable {
    @objc public init?(with format: AVAudioFormat? = nil) {
        // <snip>
        #if os(iOS)
        // This section is disabled as a workaround to avoid crashind due to:
        // "required condition is false: format.sampleRate == hwFormat.sampleRate"
//        // we have to connect the input at the original device sample rate, because once AVAudioEngine is initialized, it reports the wrong rate
//        do {
//            try setAVSessionSampleRate(sampleRate: AudioKit.deviceSampleRate)
//        } catch {
//            AKLog(error)
//            return nil
//        }

        AudioKit.engine.attach(avAudioUnitOrNode)
        AudioKit.engine.connect(AudioKit.engine.inputNode, to: self.avAudioNode, format: format ?? formatForDevice)
        // This section is disabled too. See the comment above.
//        //Now set samplerate to your AKSettings sampling rate, it may be heavy handed to make the init fail here, but taking all percautions to avoid all the hard crashes with AKMicrohpone init issues of late.
//        do {
//            try setAVSessionSampleRate(sampleRate: AKSettings.sampleRate)
//        } catch {
//            AKLog(error)
//            return nil
//        }
          // <snip>
    }

    private func getFormatForDevice() -> AVAudioFormat? {
        // <snip>
        #if os(iOS) && !targetEnvironment(simulator)
        let desiredFS = AVAudioSession.sharedInstance().sampleRate
        // <snip>
    }

The obvious downside is that a lower sample rate is used but at least the app doesn't crash when AKMicrophoneWorkaround(with: nil) is called.

Hope this helps!

[m-graf]

Hi all,

Here's how I was able to reproduce the issue reliably on iOS:

1. Launch the app
2. Play an audio file using AVPlayer. The audio file I used was sampled in 44.1kHz.
3. Call AKSettings.setSession(category: .playAndRecord)
4. Call AKMicrophone(with: nil)
5. Crash: required condition is false: format.sampleRate == hwFormat.sampleRate

The format had sample rate of 48kHz while hwFormat had sample rate of 44.1kHz.

(lldb) po AVAudioSession.sharedInstance().sampleRate
44100.0

(lldb) po AudioKit.deviceSampleRate
48000.0

I was able to fix the issue by re-implementing AKMicrophone. The code is basically a copy-paste from the original class but I removed the calls to setAVSessionSampleRate() method.

Also, I modified the AKMicrophone.getFormatForDevice() method to avoid using AudioKit.deviceSampleRate and use AVAudioSession.sharedInstance().sampleRate instead.

Here are the relevant changes in code:

@objc open class AKMicrophoneWorkaround: AKNode, AKToggleable {
    @objc public init?(with format: AVAudioFormat? = nil) {
        // <snip>
        #if os(iOS)
        // This section is disabled as a workaround to avoid crashind due to:
        // "required condition is false: format.sampleRate == hwFormat.sampleRate"
//        // we have to connect the input at the original device sample rate, because once AVAudioEngine is initialized, it reports the wrong rate
//        do {
//            try setAVSessionSampleRate(sampleRate: AudioKit.deviceSampleRate)
//        } catch {
//            AKLog(error)
//            return nil
//        }

        AudioKit.engine.attach(avAudioUnitOrNode)
        AudioKit.engine.connect(AudioKit.engine.inputNode, to: self.avAudioNode, format: format ?? formatForDevice)
        // This section is disabled too. See the comment above.
//        //Now set samplerate to your AKSettings sampling rate, it may be heavy handed to make the init fail here, but taking all percautions to avoid all the hard crashes with AKMicrohpone init issues of late.
//        do {
//            try setAVSessionSampleRate(sampleRate: AKSettings.sampleRate)
//        } catch {
//            AKLog(error)
//            return nil
//        }
          // <snip>
    }

    private func getFormatForDevice() -> AVAudioFormat? {
        // <snip>
        #if os(iOS) && !targetEnvironment(simulator)
        let desiredFS = AVAudioSession.sharedInstance().sampleRate
        // <snip>
    }

The obvious downside is that a lower sample rate is used but at least the app doesn't crash when AKMicrophoneWorkaround(with: nil) is called.

Hope this helps!

disclaimer: I'm a complete noob to Swift.

Thank you so much for this! I'm trying to implement your code changes, but I'm getting a handful of errors. Can you explain how to implement this?

Type 'AKMicrophoneWorkaround' does not conform to protocol 'AKToggleable'

[warpling]

@m-graf mind cleaning up your comment to not quote the entire previous comment?

The <snip> comments are placeholders for missing code that wasn't included in the pasted example. This AKMicrophoneWorkaround is a clone of AKMicrophone with the comments out parts removed. Does that make sense?

[aure]

Accepted the PR, this will be in an AK release later today.

[davidjaystrauss]

@aure Just to clarify, the workarounds described above are no longer necessary, right?

[eofs]

@davidjaystrauss I stopped using the workaround as the latest release contains the fix and haven't had any issues since 👍

[ChaseStas]

Can you guys help me?
I started to have even more crashes with latest update. Can't figure out what's wrong.
Crashes on AKMicrophone(). Use it in viewDidLoad

    do {
        try AKSettings.setSession(category: .playAndRecord, with: [.allowBluetoothA2DP, .allowBluetooth, .defaultToSpeaker])
    } catch {
        AKLog("Could not set session category.")
    }
    AKSettings.defaultToSpeaker = true
    AKSettings.useBluetooth = true
    AKSettings.playbackWhileMuted = true
    AKSettings.audioInputEnabled = true
    self.micNode = AKMicrophone()
    self.freqTrackerNode = AKFrequencyTracker(self.micNode)
    self.boosterNode = AKBooster(self.freqTrackerNode, gain: 0)