Surge: Portable mode ?

[geoslake]

Right now all files but the dll are stored in local appdata, which imho is not clean and adds complexity, fi when making backups or moving to another pc.
Would be nice to have the option to have the "surge" folder alongside with the dll (maybe the dll could detect that folder when its present and use that instead of appdata).

Cheers

[AtmanActive]

+1
Version 1.5.0 for x86, from 2008. is portable.
Why did we lose portability with this new version?

[esaruoho]

What does portable in this case actually mean? .exe instead of .dll?

[geoslake]

? Ehh no, see op please, surge data folder along with the dll. Im currently using symlink but native support would be nice.

[esaruoho]

@baconpaul what's your take on this one?

[AtmanActive]

On Windows, portable software means that it is not enforcing it's paths and dependencies on users.
In other words, Portable Software can have an indefinite lifespan, since it can work correctly on any windows without requiring any action from user. On the other hand, a non-Portable software is forcing the user to perform the dreaded Setup.exe over and over again on each new windows installation.
In a nutshell: Portable Software is self sufficient and intelligent enough to discover it's own working path and just keep working from there. Non-portable software depends on this or that to be present in this or that folder and is scattered all over the disk system and O/S.

[baconpaul]

My take is “I haven’t used windows for development in a long time”. Sorry!

This sounds like a good idea but I have no clue how you would do or test it.

[AtmanActive]

From a programmer's point of view, it is really simple:

• never assume any 'well known' or other common paths
• always discover your own's executable path dynamically and use that as your root path
• read/write only from/to this root path you discovered in the previous step

[baconpaul]

Oh, so basically "be like a mac .app bundle" type thing. Makes sense.

I don't have a windows build or test environment so can't do any windows dev but are there particular spots in the codebase where you see us violating these rules? May be useful for whomever ends up fixing them.

[AtmanActive]

/master/src/common/SurgeStorage.cpp: lines 193-207

PWSTR localAppData;
   if (!SHGetKnownFolderPath(FOLDERID_LocalAppData, 0, nullptr, &localAppData))
   {
      CHAR path[4096];
      wsprintf(path, "%S\\Surge\\", localAppData);
      datapath = path;
   }

   PWSTR documentsFolder;
   if (!SHGetKnownFolderPath(FOLDERID_Documents, 0, nullptr, &documentsFolder))
   {
      CHAR path[4096];
      wsprintf(path, "%S\\Surge\\", documentsFolder);
      userDataPath = path;
   }

[kurasu]

As a default, I think its better to use the established paths of the operating system for storing data, as its what the average user expects anno 2018, i.e I don't need to worry about the file system for installation and when I save something myself it end up somewhere in my documents folder.

Bu I wouldn't be against it as an option for those who prefer it, it could look for the localAppData relative to the plug-in first, and then fall back to looking it up using the shell methods above. Presets saved should still go into Documents though..

There would be some more downsides to this though:

• each plug-in type needs its own copy of the data directory, which is why I changed it in the first place
• more junk host applications need to skip when scanning you plug-in folder
• users who like their VST folders tidy might find it messy

[AtmanActive]

@kurasu : I can see that you don't understand Portable paradigm and are light years away from it.
Happy reinstalling/backing up/restoring and starting your digital life all over from scratch each time you change hardware or O/S.

I'll stick to v1.5.0, thank you very much.

[geoslake]

As a default, yes, it should follow MS guidelines. But there should be an option to have it all clean, all in one folder.
Presets in documents is a bad thing to me (even though I agree it should be default), because if you reinstall the os, it's easier to mess things up (f. I. Heard of 1809 documents story?).
As a user speaking for simplicity, my view is that nothing but the OS should be stored on c:
I think there's definitely something to be said about the beauty of having a single folder with everything inside, daw, plugins presets etc, that is even runnable from a usb stick.

Cheers

[Jorgeelalto]

Reaper also has an option to install the app as a portable app - you only need to run the setup once (for a version number, that is), which extracts the executable and all the needed folders and files on the same folder. Then you can move that Reaper folder wherever you want and it will work.

For me I also think a portable install is better but I suppose giving the user the option is the best bet.

[esaruoho]

@Jorgeelalto maybe this would be useful later when we have functioning automated 32-bit & 64-bit builds for windows (VST2,VST3 <- 64bit works now, 32bit packager might be coming soon), Linux (VST2..and..VST3?..and...?? @jsakkine ), macOS (AudioUnit,VST2,VST3 <- 64bit works now).

I get this feeling that if we try to get this to be standalone before all of those work, we're gonna run into serious issues.

[Jorgeelalto]

@Jorgeelalto maybe this would be useful later when we have functioning automated 32-bit & 64-bit builds for windows (VST2,VST3), Linux (VST2..and..VST3?..and...?? @jsakkine ), macOS (AudioUnit,VST2,VST3 <- these work as 64bits now). I get this feeling that if we try to get this to be standalone before all of those work, we're gonna run into serious issues.

Definitely, the best way to work on this now is to focus on fixing stuff. The portable thing is a minor problem IMO, given that a lot of plugins are not portable.

[AtmanActive]

I have exactly 2144 (two thousand one hundred and forty four) windows VST plugins, some VSTfx other VSTi, some VST2 other VST3, some x86 other x64, and each and every single one of them works portable.

Hence, I couldn't really agree with the statement that "a lot of plugins are not portable". As a matter of fact, windows VST plugins were portable even before the whole portable software idea started spreading on windows.

[mkruselj]

There would be some more downsides to this though:

* each plug-in type needs its own copy of the data directory, which is why I changed it in the first place
* more junk host applications need to skip when scanning you plug-in folder
* users who like their VST folders tidy might find it messy

@kurasu, with all due respect, these are all non-issues if you see how u-he does it.

u-he does this perfectly IMO. Their plugins completely don't depend on any existing paths on Windows at least. There's the data folder for the plugin (called .data) which contains all the assets, presets, databases, microtuning scales, etc. that the plugin needs. Oh, and one data folder can cover for multiple plugin types, including VST2, VST3, AAX, instrument and FX plugin variants...

This data folder can be actually placed anywhere on the system, then the plugin installer just creates a link to it (not a symlink, just a regular link) - DAWs don't follow through the links, so even if therer are thousands of files behind that link, it doesn't make plugin scanning any slower.

By default, the installer detects where your VST folder is. If it's still the nasty Steinberg's default in Program Files, then it offers to install this data folder in user's Documents/u-he/.data, because this is always a fully read/writeable folder (which the plugin of course needs, which by default you cannot have in Program Files, that folder is read-only unless you're admin user and you changed permissions on it...). If your VST folder is not in a system-write-restricted area (like, raw C: or whatever other drive), then the installer allows you to install the data folder along the plugin DLL itself.

The beauty of this approach is that absolutely everything is in that data folder. You can copy out that folder and plugin DLL clean on another machine and _it will all work perfectly_. Please look into this all, guys. It's the best solution for having your plugin portable.

[baconpaul]

Is there some code we can crib that does that on u-he world?

I would love someone to tackle this. I am just not enough of a windows dev to be confident I would get it right. But would gladly and happily review and test PRs which made these changes on windows.

Thank you for the detailed information.

[mkruselj]

I don't know honestly, but Urs is a reasonable man and I'm sure if asked he can provide info. :)

[jarkk0sakkinen]

I don't really want to try to comprehend this. Show me the code and I will look at it. Thanks.

[jarkk0sakkinen]

We should close this unless someone has something solid to show.

[baconpaul]

Let’s leave it open @jsakkine- I think on windows at least it is some research and package changes. If we get a windows dev on board with deep experience in windows packaging could be a food issue for her or him

[jarkk0sakkinen]

OK, cool.

[mkruselj]

The way u-he do this on Windows is exemplary.

Every plugin (unless it's a bundle like Zebra) has a data folder right there next to the VST2 plugin file. In the installer you can choose the location of the data folder, then a link is created next to the plugin DLL (so in this case both VST2, VST3 and AAX can refer to the same location which you chose in the location).

[baconpaul]

Ok so if we change surge so

1: the installer prompts for the asset directory but chooses %locaappdata% as default
2: the installer makes hard links next to the dll
3: the dll checks it’s cwd for a hard link before %locaappdata% and errors only if neither exist

Then surge is “portable”?

I don’t really understand the windows installer architecture well enough to do this but sounds like that’s what you are saying yeah?

Thanks!

[mkruselj]

Ok so if we change surge so

1: the installer prompts for the asset directory but chooses %locaappdata% as default
2: the installer makes hard links next to the dll
3: the dll checks it’s cwd for a hard link before %locaappdata% and errors only if neither exist

Then surge is “portable”?

I don’t really understand the windows installer architecture well enough to do this but sounds like that’s what you are saying yeah?

Thanks!

That is exactly right, Paul!

[geoslake]

Ok guys, any progress on this "issue" ?
Now its a mess to me, i have

• surgeuserdefaults.xml located in users/documents
• configuration.xml in local appdata
• a symlink in programdata (which is the same as $appdata right ?) thats points to a folder in surge's own dll folder (cos this mess has nothing to do on my windows partition imho)

So pleaaaaase guys, could you allow us to put all files related to surge in surge dll or .vst3 folder, as an option of course ? Using the "getplugindir" function maybe ?
Like, it looks first into its own folder to get all it needs, and if theres nothing there, it looks for it in appdata or wherever Microsoft decides it should live in ?

Or at least, keep that minimal SurgeUserDefaults.xml in documents folder, and let it handle more things, like wavetable location etc ?

Im super grateful for all what you guys are doing to improve Surge, its amazing, but this drives me nuts.

Cheers and thanks

[baconpaul]

Sorry it drives you nuts!

We don’t have any experienced windows developers on the team. The best thing you could do to help us solve this issue is to recruit one who would tackle this. The active developers on the project use mac and linux and don’t have experience making portable applications.

I would happily have someone who knew how to do this make the changes to the code, both the installers, the regtests, and the azure pipelines and would be glad to help them. But we need a person who knows how to do all of that t do it.

[thesobercoder]

@baconpaul Even though I am not a C++ developer but I have been a windows developer for the past 10 years and my primary expertise is in C#, but I feel I can still help you guys. I've been studying other synths such as the TyrellN6 and Zebralette and here is how they make sure that the synths work in a portable mode as well. In their installation folder they always have a symlink to their data folder which they name like TyrellN6.data and the actual folder lives somewhere in %LocalAppData% and so the DLL doesn't care whether its the actual folder or a symlink, all it cares about is that its present in the same location as the DLL itself. So I tried copying the DLL and the actual folder to my DAW VST location and it worked perfectly fine. So the idea here is not hard code paths like %LocalAppData% and always expect that the folder resides in the same location as the DLL itself. During the installation phase you can create a symlink to the data folder which can reside anywhere practically and if someone want a portable version they can just copy over the DLL and folder to a location and it will start working in portable mode. Please let me know if there is anyway I can help. I love this synth and I would love to see the portable version.

[baconpaul]

Hi @thesobercoder yeah that sounds like what @mkruselj had said also!

So the bits I don’t know how to do are

1 in innosetup make the hard link from programdata to the install location

1. In surgesrorage.cpp find the dll location and look next to it as a first guess

On Mac and Linux we check several locations and I suggest we do same on windows. That is if “dlllocation/surge” doesn’t exist try “Programdata/surge”. that logic is in surge storage.cpp

I think if you know innosetup and the Windows apis it’s not that hard - happy for you to take a look! And thanks for offering to help!

[mkruselj]

@thesobercoder Actually u-he plugins don't use symlinks, they use regular shortcuts (like those you create from the right-click context menu when clicking on files or folders in Explorer) on Windows, and aliases on Mac. Just FYI. :)

So what would be needed here is reading the destination path of the shortcut that is next to the DLL. Also, some users can actually remove the shortcut and have the data folder right in there next to the DLL instead, that should also work (so, first detect if it's a folder, then use it - if not, it's a link, then get link's destination using some API that I'm sure exists, etc.)

[baconpaul]

Right you want a hardlink not a symlink. A hardlink means the files live if you delete any of the copies. A symlink means the files die if you delete the original.

[baconpaul]

And a hard link means it really is just file exists as your api. The os will handle all the reference counts. So the work really is

Make innosetup install into program files
Make innosetup out a hard link beside the dll
Make the dll resolve its path and look side by side for the surgendir
If it isn’t there look in program files

[K0rrid0r]

I know you guys havent used surge 1.5.2 but in that installation all the .dll, presets, .wt files was installed in one directory.

But when i started to use windows 8.1 this directory + that appdata folder was created, took me a while to realise new presets were not saved into the datafolder where the .dll files were but instead in appdata.

It's a mess. but once you know it you can handle it and do loads of paths to go around it and have oversight.

So in that sense i like how its structured now.

[baconpaul]

so with hard links it would still kind of be that way (namely the data would be in programdata) but it would also be next to your vst2 and vst3. The reason it moved to a common area was to deal with multiple plugin flavors, I think (and also to avoid 3 copies). So a proper portable install on windows, as I understand it, is "the best of both worlds".

But what do I know. I haven't used windows for anything in 5 years other than VM builds, and haven't done windows dev in a decade!

[geoslake]

Guys im no programmer but i dont understand why its so complex. Every developer i talked to about this, solved it by using the "getplugindir" (or "getpluginpath", not sure) function.
You can make your installer install data, presets, preferences etc in whatever appdata, documents, roaming nonsense you want, as long as this function is implemented and the plugin is given a chance to look in its own folder for its data, its fine. Then, "advanced" users just copy the required files there, done.
Mike from Audiorealism literally did it in 10 minutes, same for the Cytomic guy and a lot more, who implemented it per my request.
Now again, i know nothing, and this function is maybe not present in the framework you use ?

Cheers & thanks

[baconpaul]

So basically just check side-by-side first. Then let the user make the folder if they want? That's it?

I guess from googling someone could just add https://stackoverflow.com/questions/6924195/get-dll-path-at-runtime as the first thing to test https://github.com/surge-synthesizer/surge/blob/7dd8ae096fb129b13a452019808133e3349175be/src/common/SurgeStorage.cpp#L253, then update the documentation and test it on our various oses and bits and push it to a pull request!

[mkruselj]

The installer should have an option for placing the data folder. By default it shouldn't be in the VST folder, because with lots of files it actually makes plugin scanning slower in certain hosts (I had this problem with Zebra and 6000 presets, scanning was so slow in Reaper because it scanned everything in VST folders, not just DLLs).

Default location for the factory stuff offered by the installer should indeed still be ProgramData. But, the user should be able to change this path to whatever, EXCEPT Program Files because by default that folder is not user-writeable since Vista onwards. The installer should just place a hardlink next to DLL to that user chosen folder path (and warn if the path is not legit).

Guys im no programmer but i dont understand why its so complex.

Because you're not a programmer, maybe? :) And also, @baconpaul is not a Windows programmer so it's not his neck of woods to worry about.

[baconpaul]

Yeah I think that's it

I suppose what @geoslake is saying is we don't even have to do that. If we just check for the surge directory next to the dll then the user can do what they want. The stack overflow I posted above would let you try that. Then move your assets as you see fit to either %PROGRAMDATA%/Surge or (path-where-your-dll-is)/Surge

@geoslake the thing you could do that would be helpful is: help us recruit a person who is a C++ programmer and knows windows well to this project. Mac Devs (in this case, me) made everything work on mac in the various packagers; Linux devs (and also FreeBSD devs!) made everything work on linux the same; but a windows dev is the thing we don't have. I stumble along in a VM and make sure that things work since the code is mostly portable, but non-portable edges like this are places where we could use help.

[baconpaul]

Didn't get it working but since I was in the installer I took a peek

Something like this

[Run]
Filename: "{cmd}"; \
    Parameters: "/C mklink /D /H ""{commonappdata}\Surge"" ""{cf}\VST3"""; \
    Description: "Install Surge portable by hard linking resources to VST3 Directory"; \
    Flags: shellexec postinstall unchecked

is very close to making the hardlink. For some reason it doesn't make the link but it does prompt me. Next time I'm in the installer I'll see if i can make it work.

[baconpaul]

Inasmuch as I understand it, I have implemented what you all describe as a portable mode
for surge. Here is exactly what happens

1. The SurgeStorage startup path will find the dll name
2. It will replace the dllname string ".vst3" with "Data" so
   c:\program files\common\vst3\Surge.vst3 -> c:\Program Files\Common\VST3\SurgeData
3. If that is a readable directory, that is your data path for presets, configuration, etc
4. Otherwise we fall back to c:\ProgramData\Surge
5. Otherwise we fall back to %LOCALAPPDATA%\Surge
6. Otherwise we fail to startup
7. Moreover the installer gives an option to make a junction from
   c:\programdata\Surge to {cf}\VST3\SurgeData at the end of the installer

When I merge that change I am going to close this issue. If the above behavior does not constitute a "portable" app, please reopen it with a clear definition of which step is wrong. And of course if the software doesn't do what I describe above, please also reopen it.

Should be in the nightly in about 20 minutes. Testing appreciated. Remember, the about screen will show you your data path when you launch.

Thank you

[AtmanActive]

[baconpaul]

Well please do test it. Entirely possible I got it wrong. New nightly should be up in 30 minutes or so.

[baconpaul]

Oh I just realized: My change won't work for the 32 bit version just the 64. (I made an assumption about dll name). So let me reopen this to tweak that too. But I'll wait to do that tweak until I have a confirm from someone who understands windows that it is correct in 64 bit vst3.

thanks

[baconpaul]

I actually just pushed the fix that will make it work for the vst3 and also the FX bank. So closing again. cc45dd57 is the hash to look out for. Probably another 30 mins. Lemme know if this isn't right.

[baconpaul]

Chatting with @mkruselj and he had some ideas about how this could be better, especially for user documents. The commit above should let him see how to make tweaks to the installer and runtime now the hard parts are done (namely: we can make the installer run extra steps; and we look up our DLL name to make paths). So happy to have tweaks here to make it less irritating to set up. Will leave it in his capable hands.

[mkruselj]

Please don't say they're capable, they're hacksaws :)

[AtmanActive]

Can confirm. Working as expected now. Great job! Thank you everyone.

[baconpaul]

Wonderful thanks for testing

[baconpaul]

Ha so against my will I now understand what a portable app is, inasmuch as it is defined! Chuckle.

@mkruselj helped me understand we needed one more tweak here which is the user data, currently stored in %DOCUMENTS%\Surge. So as well as looking for factory data in (dllFolder)\SurgeData first, Surge will use (dllFolder)\SurgeUserData if it exists preferentially over %DOCUMENTS%\Surge. It's up to the user to set up that folder if they want.

Pushing this morning, and then this one is finally done and dusted.