Lmms: 64 bit VST on Linux/Wine

[zonkmachine]

Wine supports 64-bit since wine-1.2 but it doesn't seem to be working under linux/wine/lmms.
https://github.com/LMMS/lmms/issues/4102, https://lmms.io/forum/viewtopic.php?f=7&t=26647#p51235

• [ ] Assess the status/quality of 64-bit VST on Linux/Wine
• [ ] If possible, activate 64-bit VST on Linux/Wine
• [ ] Warn if needed/possible when running 64-bit VST on Linux/Wine
• [ ] Update documentation

---

Wine 3.0 is in the works and rc5 was recently released.
https://www.winehq.org/

[lukas-w]

I did some work to make building 64-bit RemoteVstPlugin work on Linux in the msvc/vst branch, but it fails loading VSTs with the message

RemotePlugin::DebugMessage: mainEntry procedure returned NULL

[lukas-w]

Fixed the error via 631a3dbcfb37f59541cc4ebab4659728fb8aeae8. 64bit VSTs load perfectly now when hard-coding LMMS to always use the 64-bit version of RemoteVstPlugin.

Unfortunately, there's still something going wrong with the trial-and-error approach we're using for starting the right version:

https://github.com/LMMS/lmms/blob/631a3dbcfb37f59541cc4ebab4659728fb8aeae8/plugins/vst_base/VstPlugin.cpp#L100-L109

The IdVstBadDllFormat message isn't always received for some reason and even when it is, starting RemoteVstPlugin32 after RemoteVstPlugin64 has already been started makes LMMS hang indefinitely.

[tresf]

@lukas-w if we do a quick file read on the DLL, we should be able to tell if it's 64-bit or not without the trial-and-error approach. https://stackoverflow.com/a/35418180/3196753.

[lukas-w]

@tresf Thanks for sharing. That would work around the issue and I think we should implement it, but I suspect some deeper IPC bug causing this that may resurface, so I think this is worth investigating first.

Update: IdVstBadDllFormat isn't received because LMMS stops processing messages from the plugin as soon as the plugin quits. This was partially fixed in 4fd8ecd7e4084aedbd39900ffe5cfd107ceba1a0 and is fully fixed in 9db8cbfb31f9b6d3e754887a6280770e3d3e2bcd. Even after that fix, LMMS hangs, which is due to missing cleanup in VstPlugin.cpp when re-starting. This is too much work fixing so I implemented the approach suggested by @tresf via 9db8cbfb31f9b6d3e754887a6280770e3d3e2bcd.

[Spekular]

Closing and consolidating into #5433