Lmms: SF2 hangs on open (Apple)

[tresf]

Software freezes and causes the user to "Force Quit" when opening any project with at least one soundfont .

• This crash occurs with SF2 files as relative OR absolute paths.
• This crash occurs with SDL OR PortAudio.
• This crash DOES NOT occur when using Dummy Audio.

The soundfont player works if added manually, but when reopening the project, the software hangs on a dialog that says "Opening project" usually around 70-80% with the spinning wheel which never goes away.

The screenshot below (sorry, not ver descriptive) is the screen that the user will see when the freeze occurs. The default project is loaded in the background while the "Opening project" screen appears and then the pinwheel appears and the software has crashed.

Possibly related: http://www.juce.com/forum/topic/coreaudio-possible-deadlock-when-moving-mavericks

Backtrace:
https://gist.github.com/tresf/230b20f083403a11a3c7

[mikobuntu]

Im not sure if this is related, but everytime i load a soundfont on linux i get this ( non fatal ) warning CRITICAL **: fluid_synth_sfont_unref: assertion `sfont_info != NULL' failedperhaps related to the fluidsynth lib files? or within LMMS itself?
anyway great job so far on getting LMMS this far forward on OSX ....
I seen from your vid that you are having a lot of trouble with the LADSPA plugins also.
thanks Mikobuntu ;)

Date: Sat, 26 Apr 2014 09:26:22 -0700
From: [email protected]
To: [email protected]
Subject: [lmms] Projects containing soundfonts hang on open (Apple) (#649)

Software freezes and causes the user to "Force Quit" when opening a project with Soundfonts.

The soundfont player works if added manually, but when reopening the project, the software hangs.

—
Reply to this email directly or view it on GitHub.

[tresf]

I seen from your vid that you are having a lot of trouble with the LADSPA plugins also.

We've omitted some items for now due to some incompatible coding standards. I haven't dove too much into those messages yet.

Edit: The LADSPA stuff has since been corrected and pushed upstream to Steve Harris's GitHub repo. It was the same _init() and _fini() bug that affected the tap plugins.

[tresf]

Some pthread specific information for OSX which may be useful:
http://stackoverflow.com/questions/15590865/crash-in-pthread-specific-on-mac-os-x

[tresf]

A _slightly_ better stacktrace containing more debug symbols:
https://gist.github.com/tresf/230b20f083403a11a3c7

[tresf]

@mikobuntu after some observations of outstanding SF2 bugs, I do believe this is related to the errors you are seeing. It appears something is not initializing properly but I still cannot find it. Will continue searching. :)

[tresf]

Possible duplicate of this bug report (stack overflow)
http://stackoverflow.com/questions/8471546/strange-fluidsynth-assertion-settings-null-result

[tresf]

Another bug report from 2006 which claims to have been resolved with a source code update.
http://lists.puredata.info/pipermail/pd-list/2006-04/036979.html

Seems to crash on a similar mutex lock area while initializing the fluidsynth settings. Interestingly enough, I did a similar snippet with if (m_settings != NULL) and and delete_fluid_settings(m_settings) prior to the initialization and was able to crash LMMS (Linux), but it won't crash on debug, so I can't pinpoint the exact crash point/error.

@tobydox how would we go about trying out some of the upstream patches? Would this cause issues on the OSs which bundle fluidsynth (i.e. Linux)? Mostly concerned with #1238 although Mac support would be an added bonus if the bugs are related.

[tresf]

Unfortunately, commit https://github.com/LMMS/lmms/commit/240c0b20a63726cebeb5177db84031428fef1cb9 does not fix this bug. Apple SF2 support will remain disabled on stable-1.1

[softrabbit]

Had a look at the backtrace (from 2014-06-05), and... why is libfluidsynth calling into CoreAudio? AFAIK it shouldn't be doing any audio output to anywhere besides LMMS.

frame #7: 0x00007fff92fb1bb6 CoreAudio`AudioObjectGetPropertyDataSize + 218
frame #8: 0x0000000109c0119c libfluidsynth.1.dylib`get_num_outputs + 76
frame #9: 0x0000000109c0135f libfluidsynth.1.dylib`fluid_core_audio_driver_settings + 303
frame #10: 0x0000000109c2891e libfluidsynth.1.dylib`fluid_audio_driver_settings + 462
frame #11: 0x0000000109c04bfb libfluidsynth.1.dylib`new_fluid_settings + 91
frame #12: 0x0000000109bc4976 libsf2player.so`sf2Instrument(this=0x0000000101b2d000, _instrument_track=<unavailable>) + 2054 at sf2_player.cpp:110

[tresf]

Well, I'm not explicitly linking against any CoreAudio files in the bundle, so I think it's a problem with a mutex lock, but I'm not very good with debugging these things...

The closest thing I could find for an explanation is here:
http://stackoverflow.com/a/8472145/1466873

I don't know if it matters, but this doesn't crash every time if the instrument is added manually. I can get the instrument to add to a track and get SF2 playback about 50% of the time.

However on project load, it crashes each time.

Also, if I change the output device to DummyAudio, it doesn't crash either.

-Tres

[softrabbit]

This looks like an identical crash: http://www.juce.com/forum/topic/coreaudio-possible-deadlock-when-moving-mavericks

So, SDL or PortAudio uses CoreAudio, and when libfluidsynth goes about using it as well, a deadlock happens if the timing is right. I'm leaning strongly towards this being a libfluidsynth bug.

Think I'll file an issue for this over at SF.net and see what the Fluid people have to say on the subject.

[diizy]

On 01/16/2015 08:12 PM, Raine M. Ekman wrote:

This looks like an identical crash:
http://www.juce.com/forum/topic/coreaudio-possible-deadlock-when-moving-mavericks

So, SDL or PortAudio uses CoreAudio, and when libfluidsynth goes about
using it as well, a deadlock happens if the timing is right. I'm
leaning strongly towards this being a libfluidsynth bug.

Hm... yet another reason to roll our own soundfont player?

[tresf]

This looks like an identical crash: http://www.juce.com/forum/topic/coreaudio-possible-deadlock-when-moving-mavericks

Hey!... That same bug is in my original post. :)

[softrabbit]

Hey!... That same bug is in my original post. :)

Right, must've been a case of scrolling blindness on my part.

[tresf]

Right, must've been a case of scrolling blindness on my part.

or TL;DR on my part :cake:

[softrabbit]

@tresf, I never got around to filing an issue, but revisiting this now I have an idea. Could you try adding a line:

fluid_settings_setstr(m_settings, "audio.driver", "file");

right after new_fluid_settings() in the constructor for the sf2Instrument ? That should prevent Fluid from defaulting to CoreAudio.

https://sourceforge.net/p/fluidsynth/wiki/FluidSettings/

[tresf]

:+1:

[tresf]

Ok.. I took the line back out and it's not crashing... I'm not sure how conclusive these findings are now. :crying_cat_face:

Edit: I installed on native hardware and the bug still exists with the recommended patch, still hangs at 83%. Not sure why the Lion VM doesn't encounter this crash (I would suspect it's sound driver related). Also, adding the instrument manually seems to be stable, it's project load that is causing the reproducible crash.

[softrabbit]

Not sure why the Lion VM doesn't encounter this crash (I would suspect it's sound driver related).

Me too. And that's the direction the earlier backtraces point as well.

Also, adding the instrument manually seems to be stable, it's project load that is causing the reproducible crash.

Can you load a project containing only one SF2 track or does that crash too?

[tresf]

That crashes too. It's technically an "infinite hang" symptom that occurs and it happens regardless of whether or not an actual SF2 file is loaded into the instrument or not.

[tresf]

Think I'll file an issue for this over at SF.net and see what the Fluid people have to say on the subject.

@softrabbit do you think this is still worth an upstream bug report? If this is an upstream issue, I'm a bit confused that other people aren't experiencing it too.

Edit: Interesting mailing list conversation here, but appears to be an unrelated 32-bit lib problem: https://lists.nongnu.org/archive/html/fluid-dev/2013-08/msg00024.html

[tresf]

Out of curiosity, I changed the modality of the progress dialog that appears when the track loads, and the frees occurs at 20% instead of 83%.

This is a shot in the dark, but is there a chance that fluidsynth is honoring some unorthodox blocking somewhere (such as on the GUI thread?). This is such a strange bug. Many of the older fixed bug reports mentioned issues with glib threads (which AFAIK we don't use), so the plugin makes some null thread reference internally to prevent deadlocks. Not sure if there's something correlated here, but the fact that is freezes 100% of the time on project load, but only occasionally after load makes me wonder if the lock is actually occurring somewhere else.

[tresf]

Finally found the JUCE commit that addressed this lock problem.

https://github.com/julianstorer/JUCE/commit/26828713ba418f0271e314c214888c1c0f1c4ea1#diff-80c212bebcf9de54a837d1e31270ca70R371

The problems seems to stem from a lock on CoreAudio (CoreAudio HALB_Mutex::Lock()).

Chromium suffered a similar problem here. The description is identical to what's the the backtrace. I don't believe Chrome ever fixed the problem, they just removed the feature from Mac.

The description:

Tell OSX to manage its own property listener callback thread.

Seems OSX might be expecting us to only make AudioObjectPropertyData
calls on the same thread as listener callbacks are fired on. Setting the
kAudioHardwarePropertyRunLoop to NULL tells OSX to manage its own thread,
which will hopefully ensure it has to fire callbacks in a concurrency safe
manner.

Another option might be to set the run loop to the one used by the
audio thread. However, currently we call AudioObjectPropertyData in
many places, not all of them on the audio thread.

In the WebRTC code base they have this code: [[...](https://bugs.chromium.org/p/chromium/issues/detail?id=158170#c33)]

Can someone point me in the right direction? Does this need to be fixed inside AudioPortAudio.cpp or PortAudio itself? Where can I look to handle the SF2's plugin initialization in a thread-safe manner?

[jasp00]

This looks like a FluidSynth problem. Generic initialization triggers CoreAudio initialization. Since we do not need CoreAudio for the plug-in, this should be avoided, but there is currently no way unless we build FluidSynth without CoreAudio.

[tresf]

This looks like a FluidSynth problem.

I hope it can be circumvented if that statement is true. FluidSynth has been stale for almost 2 years which puts this bug in a bad position.

Since we do not need CoreAudio for the plug-in, this should be avoided

I'm interested what drives the assumption that CoreAudio is not needed for the plugin on Mac. If the soundcard is switched to Dummy Audio, the bug goes away, so there's something it uses from CoreAudio.

Furthermore, instantiating the plugin from the GUI it crashes occasionally whereas on project load it crashes every time. Are there any techniques that can be used to force synchronization for this plugin loading when on Mac?

[jasp00]

FluidSynth has been stale for almost 2 years which puts this bug in a bad position.

Their mailing list is active. Do we wait until fixed versions are released?

I'm interested what drives the assumption that CoreAudio is not needed for the plugin on Mac.

The plug-in does not need to talk to any audio driver on Mac, Linux or other OS.

instantiating the plugin from the GUI it crashes occasionally whereas on project load it crashes every time.

These look like different problems that should be addressed separately.

Are there any techniques that can be used to force synchronization for this plugin loading when on Mac?

Do you mean that the plug-in should use some sort of synchronization with CoreAudio? The plug-in should not call any audio driver.

[tresf]

The plug-in should not call any audio driver.

Repeating the same words over and over without explaining why doesn't help. From everything I'm reading as well as previous bug-hunting on Mac, this is a threading problem causing the crash. Whether or not it SHOULD be talking to CoreAudio is an architectural question about the library -- one I'm not going to tackle. My question was really hoping for a threaded work-around.

Their mailing list is active. Do we wait until fixed versions are released?

We normally get fluidsynth from repos, so yes. Although it is good to see activity on the fluidsynth mailing list, the state of the bug tracker is not as promising. For example... a basic MSVC++ pull request has been open for 2 years.

Anyway... Homebrew can handle custom formulae so if we do fork and patch this library, it would be trivial to shim the steps into our build procedures. I'm not familiar enough with the library to write this patch, but I'd be very happy to write the procedures around it.

[tresf]

These look like different problems that should be addressed separately.

In both instances the crashes occur during plugin initialization.

Since the crash occurs due to a race condition with the mutex inside CoreAudio I believe the crashes to be directly related to each other. According to the JUCE article, the callback which returns after getting the hardware information makes an assumption about what thread it is being run on.

I'm not an expert on threads in C++, but I've seen these crashes happen with other apple calls in other projects and they generally come down to threading, fixed by calling everything from the same thread.

[jasp00]

The plug-in should not call any audio driver.

Sf2 Player should not call to CoreAudio, ALSA, PortAudio, etc. Plug-in should deliver data to LMMS core, which is the one that calls CoreAudio, ALSA, etc. If LMMS core locks CoreAudio, Sf2 Player will hang if it tries to lock CoreAudio too. I do not understand what a threaded work-around is supposed to do.

if we do fork and patch this library

You only need to configure with --disable-coreaudio.

[tresf]

You only need to configure with --disable-coreaudio.

Fixes the crash, but the command is off a bit...

# download and extract fluidsynth
mkdir build target
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../target -Denable-coreaudio=off -Denable-framework=off
cp ../target/lib64/libfluidsynth.1.5.2.dylib /path/to/LMMS.app/Contents/Frameworks/libfluidsynth.1.dylib

INSTALL_PREFIX is for obvious reasons, but the disabling the framework is required to obtain a dylib rather than the default behavior of installing it to /Library/Frameworks/FluidSynth.framework (which doesn't have permission anyway).

I'll see if I can script this into the installation script and finally close this out.

[tresf]

PR is ready. https://github.com/LMMS/lmms/pull/3446

[derselbst]

Please consider using fluid_audio_driver_register() for a runtime solution which audio drivers to initialize during new_fluid_settings().

[tresf]

@derselbst thanks. This feature is new with 1.1.9 so I think we'll have to do some version detection. Is it sufficient to simply pass an empty array?

[derselbst]

You can use the FLUIDSYNTH_VERSION_* macros.

Is it sufficient to simply pass an empty array?

In order to deregister all of fluidsynth's audio driver, yes:
https://github.com/derselbst/ANMP/blob/237e373c2756e218322600468ed9e7fc6eea6f3c/src/InputLibraryWrapper/FluidsynthWrapper.cpp#L237-L239

[tresf]

Thanks, we'll get this added.

Edit: PR is ready: #4300