Rack: Version upgrade pain mitigation

[AndrewBelt]

When Rack updates to v0.6, plugins will need to update their source if needed, recompile, repackage, reupload, and update their manifest file in the community repo. I estimate 10% will do this on the first day and 25% in the first two weeks. The lack of plugins will persuade users to not switch to v0.6, which will make the transition very difficult.

Here are my ideas:

Offer a build service for plugins

Many plugin developers have dropped support for their plugins, yet users will still want them to work.
I don't have the time for building plugins myself, so I would need volunteers.

Volunteers would need to maintain forks for each plugin if source changes are required.
This will require less than O(n) work, because once a build system is prepared for an architecture, it's easy to git pull, make dist, and upload everything at once.

Adopt a "release candidate" versioning system

Ideally this should be called "beta", but v0.x is already the beta version to v1, so the release candidate could be called v0.6.0rc1 or something.

This also splits the userbase in half, which I mentioned above that I want to avoid, so it will only be available for 1-2 weeks until v0.6.0 is released.

Rack v0.5 Plugin Manager will no longer update plugins after v0.6 is released

There should be no need, since no plugin should release v0.5 updates after Rack v0.6 is released, and loading a v0.6 plugin would not work in Rack v0.5.

Call the next release Rack v1 instead

This would be a good excuse for breaking compatibility of plugins.
However, the excuse can be used a limited number of times.
Therefore, a well-designed upgrade method should be planned for future releases.

[jeremywen]

The main issue I see is contacting devs and figuring out who isn't going to do it themselves. I like the fork and build service idea.

[jimm]

Such a service would also be nice for devs like me that don't have access to Windows, Linux, and/or Mac OS boxes. It could double as a continuous integration tool of sorts.

[mdemanett]

What if there was a way for devs to get ready ahead of the day?

So when you say it's time, we can all go build against 0.6, and post a compatible release, without yet withdrawing the 0.5 builds.

The plugin.json format would need to support versioned downloads, something like:

{
  ...
  "downloads": {
    "0.6": {
      "win": { ... },
      "lin": { ... },
      "mac": { ... }
    },
    "0.5": {
      "win": { ... },
      "lin": { ... },
      "mac": { ... }
    }
  }
}

If the announcement goes out say two weeks before release time, active devs have time to get their stuff right. Then, say a couple days before release, for whoever isn't up to date, maintainers can fork and fix. On release day end users should have a seamless experience.

Maybe those version strings should be like "0.5.x", "0.6.x", where x the module internal version, to match the module version scheme, and the top-level version field in the json goes away.

[jeremywen]

@AndrewBelt Do you think if we allow forks in the community repo, we should have a "builtBy" kind of property?

[s-ol]

Offer a build service for plugins

in #509 I presented a way of cross-compiling to MinGW/Windows from (Arch) linux and a docker image to do this (as of v0.5). With a rather small amount of effort this should be integrate-able into a Circle- or Travis-CI build process that creates linux and windows builds. Some CI platforms offer Mac VMs too but I have no experience in building there.

[jeremywen]

it's more about upgrading code though not really a build problem

[AndrewBelt]

Let's assume I'll give 1 week between the release candidate and the public release.

@jeremywen I think a reasonable protocol is for me to write a stock message to be posted as an issue on every open-source plugin repo, informing them that they need to build for the new version. If nothing happens after 4 days or so, the build team can spend the next 3 days adopting their plugin.

@mdemanett It might be best to actually just use a separate branch called "v0.6" for the community repo.

@jeremywen Also, where is a good place to host all these build team plugins?

@s-ol It's easy for a build team to just use a normal build system. They'll need to test on each OS anyway.

[jeremywen]

The builds could just be inside the github releases page inside the forked repo just like most devs normally do for their own stuff.

[AndrewBelt]

That's a good method for releasing your own plugins, because each "release post" equals one version. But for N plugins, something like

for p in plugins/*; do (cd $p; make dist; rsync $p/dist/*.zip some_server:); done

would be easier.

[jeremywen]

Maybe I am misunderstanding then. I was thinking the person volunteering is sort of adopting a neglected child 😄 and taking the modules in to build them and keep them updated just like they would their own.

[AndrewBelt]

Oh, I see. Actually, I think v0.6 won't require many API changes, so it might suffice for someone on the build team for each architecture to

• Clone all repos
• Do a lazy search-and-replace for reset -> onReset, etc.
• Build everything

If the plugins don't work still, then adopting them might be necessary.

[jeremywen]

I was just trying to think ahead also, like what about 0.7 and 0.8 and so on...

[AndrewBelt]

The next versions are v1, v2, and so on, and as I mentioned, this next version might be called v1 instead of v0.6.

There will become a time where unmaintained plugins need to die. Those who want to adopt them definitely can, but they should be willing and not feel pressured.
So yes, if a plugin developer doesn't respond in 4 days, the build team should just try to build a v0.6 compatible version. If they don't respond for a month, someone can adopt them permanently.

[jeremywen]

Ah I see now.

[jeremywen]

I guess the community repo could have a releases page for these ones we want to quickly build and upload.

[djpeterso23662]

The scripts for changing from the 0.4.0 API to the 0.5.0 API were hugely helpful. So putting out some conversion scripts and giving them a high profile mitigates many of the issues mentioned above. A long-form article on the API changes will also help.

[djpeterso23662]

A small suggestion: developers could be required to provide a reliable communication channel in order to be in the package manager.

[senisioi]

For 0.6 there's not much to do than to have some script to do replacements for the plugins. Maybe another solution would be to keep the functionality backwards compatible across a few versions and throw some warning that some things will become deprecated after 0.X so that new modules don't make use of it. And you could have a plan to deprecate completely something after some major release.
There's a lot of work being done, which is great, but it's hard to keep the plugins up-to-date version after version.
The adoption thing sounds like a nice solution for future releases.

[AndrewBelt]

@djpeterso23662 In an idea world, everyone should subscribe to #258, but one of us will post on every GitHub thread just in case.

@senisioi Hmm, I could deprecate reset and randomize. But they'd still need to be recompiled to match the ABI of v0.6, so I might as well remove instead of deprecate since it's an easy reset() -> onReset() and randomize() -> onRandomize() search-and-replace.
So now, it's shouldn't be "hard to keep the plugins up-to-date version after version" because it's two search-and-replaces and a compile, that's it.

[djpeterso23662]

@AndrewBelt are we now at a point where building the latest Rack source enables v0.6.x-compatible plugins to be tested? If not, would you guess we are days, or weeks, from that point in time?

[AndrewBelt]

I wouldn't worry about it until January 31 or so.

[simonvanderveldt]

Offer a build service for plugins
Many plugin developers have dropped support for their plugins, yet users will still want them to work.
I don't have the time for building plugins myself, so I would need volunteers.

Volunteers would need to maintain forks for each plugin if source changes are required.
This will require less than O(n) work, because once a build system is prepared for an architecture, it's easy to git pull, make dist, and upload everything at once.

Would it be an option to create a plugin skeleton/template with a default CI config that does the builds and releases it on GitHub? That way more plugin devs would automatically build releases for their own plugins from the start.
Obviously that doesn't address the situation where someone stops working on a plugin they created but it would at least ensure there are always builds for the code in it's current state.

Note sure if it should be part of this issue, but out of tree builds would be good as well, it's quiet impractical for plugin CI to need to build within the Rack tree. Just a header and linking to the lib(s) should be enough.

The lack of plugins will persuade users to not switch to v0.6, which will make the transition very difficult.

Depending on the API changes it might be less work to be backwards compatible (as mentioned above) with version n-1 than to fork a whole bunch of repos and update them. The list of repos is probably not going to get shorter over time so this balance will only get worse.
Being backwards compatible would ensure there's never a reason for a user to use version n-1. On the other hand it might de-incentivize the plugin developer to update the plugin. Choices choices...

[AndrewBelt]

The problem with CI is that it doesn't include a human to test the plugins or ZIP packaging on other architectures (which has been typically a 10% build or runtime issue on other architectures). CI is also expensive in $$$$ for commercial plugins, so it would simply not be used in that case.

Can you describe more of what is needed for an out of tree build? My first thought is that the include ../../plugin.mk would need to be include $RACK_PATH/plugin.mk or something, which gives an additional difficulty of defining an environment variable when building a plugin. Describe your ideal folder structure.

As for API changes, I'll re-add deprecated methods Module::reset() and Module::randomize(), but there might be more changes that can't be backward compatible, so someone would have to fork anyway. We'll see if Rack v0.6 has any more breaking changes.

[djpeterso23662]

Hi Andrew! I took your advice not to "worry about it until January 31 or so." So I thought I should check in and see if you planned to push v0.6.0 this week, or if we are still a couple weeks or so away. What do you think?

BTW, I really enjoyed your interview on Darwin's podcast. https://www.youtube.com/watch?v=VrlI06FnAGU Really good discussion.

I bet there has been VCV Rack buzz at NAMM this weekend!

[AndrewBelt]

Still probably 3-4 weeks away unfortunately. Commercial plugin first, then back to v0.6 development.

[djpeterso23662]

Okay, thanks Andrew! I have plenty to do as well, so I am glad to wait!

[cschol]

I had my vcv-plugindownloader script clone all plugins that have source on GitHub and build them against v0.6dev tonight (on Windows).

Out of a total of 78 plugins, 57 had source available.

Here are a few things I noticed when looking at the errors that might be worth mentioning in #258 (API Update Thread):

• Sample Rate Converter API changes (setRatio, setRatioSmooth), also removal of samplerate.h
• components.hpp -> componentlibrary.hpp
• math.hpp -> util/math.hpp
• removed util.hpp, replaced with util/[...]

Most of the changes look pretty straight forward when using the API Update Thread as a reference.

[AndrewBelt]

• I'm surprised anyone is actually using samplerate.hpp.
• There is no need to explicitly include any of those. They are automatically included with rack.hpp

[cschol]

• I found samplerate.h multiple times, which was from the old sample rate library.
• Cool, that makes things easier. Developers just need to know to just include rack.hpp and remove the other includes instead of searching for replacements, which might cause issues the next time around.

[AndrewBelt]

Closing as this dicussion is no longer relevant/useful.