First of all - I love this project, this is the greatest open source audio project since Audacity. I think this is going to seriously change electronic music.
One thing that I'd lovvvvve to have for Rack is the ability to browse and install new modules from within Rack. Currently it requires a fair bit of digging all over the internet to find good user-made modules that are built for the right architecture, etc.
I think it'd be awesome if there were something like Atom Package Manager or NPM, etc, but for VCVRack - vpm maybe? Then I could publish my own modules and play with other users's modules, etc.
Anyway, this is a pretty big ask, but I think it could be a huge boost for the project. Would love to know your thoughts!
A snarky note: I can really tell who's coming from the HN crowd from yesterday. :stuck_out_tongue:
You're welcome to proceed, but as the name VCV is in the process of being trademarked, please inform me of the methods you choose to use the name.
However, I should note that your efforts will be overwritten in 2-3 months as I begin to accept third-party plugins to the "VCV store/website". The first batch will be in a week or two.
I'll have you know that I've been using Rack since 0.3! HN just let me know about the new 0.5, which is coming along very nicely!
I'm certainly not trying do to anything to "side-step" your copyright, I really just want a way to publish my own modules and install other user-created modules. Since it seems like you've already had that on the roadmap, I think we can close this, although I would like to link it to #445 since I think designing a useful manifest file for modules could make some of these easier for everybody.
Reopening this because I've changed my mind. Instead of myself maintaining only a few select third party plugins on the VCV store with a high standard, it would be a better idea to collect all plugins into a single repository with the only standard that it is not malicious software or an IP violation. Plugins with a high enough standard will be featured on the VCV store in a rotation.
I will create it under the VCV Github organization, and you and a few others can be a maintainer if you'd like.
In the other thread, you mentioned that plugin packages should contain a manifest file with its own metadata. I don't think plugins themselves should include this information, but the package manager software (built into Rack itself?) should "own" them in the same way that Linux package managers own their packages on the filesystem. In Linux packages, the software (upstream) doesn't include metadata about itself, but the package manager software keeps a copy of all the available plugins and all installed plugins.
So let's decide on a JSON manifest format in this thread, and the package manager software itself.
(Will update this post as changes are suggested.)
File structure of repo (can't decide on a name, "community"?):
Each JSON package file should look like the following.
{
"slug": "AudibleInstruments",
// optional, if not present, name is set to slug
"name": "Audible Instruments",
// optional, let's not parse this against a dictionary of licenses or anything, just display the verbatim text as needed
"license": "BSD 3-clause",
// Latest version, without the "v" prefix
"version": "0.3.0",
// optional
"homepage": "https://vcvrack.com/",
// optional
"manual": "https://vcvrack.com/manual/AudibleInstruments.html",
// optional
"source": "https://github.com/VCVRack/AudibleInstruments",
// ZIP file should contain a folder equal to the slug or your entry will be removed
"download": "https://example.com/AudibleInstruments-0.5.0.zip",
// used only if your plugin is sold through the VCV store
"productId": "1234567890",
// optional, if not present, all are assumed
"arch": ["win", "lin", "mac"],
// optional
"sha256": "5993dbe15af246092678e00897dc7a4a1e2dabc494b83412fe31200d5bb58305",
}
Note that there is redundant information in plugin.hpp. I'd like to get rid of as much as I can and perhaps rely on the package manager to provide this information as needed.
Should it handle multiple release urls if you want to split builds for each OS? No, not supporting this gives an incentive for developers to properly package their builds. Someone should write a zipmerge.sh script for the community that actually works.
Should it support old versions for the hell of it? No package manager I'm aware of does this, would be too messy and too time consuming for the plugin developer.
Rack itself will handle the downloading of plugins through api.vcvrack.com, as it currently does with official VCV store plugins.
This VCV server will keep a synchronized copy of the community repo and convert the JSON data to an API.
Installed packages should be associated with your VCV account. (Yes, not the favorite for maybe 0.2% of users, but they always have the option of manually downloading the plugins from the download URLs and placing them in the plugins/ directory.)
This allows the package manager selection to be web-based on https://vcvrack.com/
If you click "Add" or "Purchase" on the web package manager, Rack will highlight the "Refresh Plugins" button to alert the user that updates are available.
The logic for determining whether the "Refresh Plugins" button is highlighted:
We need a way to handle download links to premium plugins.
The download link for premium plugins obviously cannot go in the manifest, so I suppose I could handle each of these as a special case internally through my checkout system.
Possible implementation notes:
Two-levels of feed. First is the subscription, second is the package. Support multiple subscription feeds. Subscription feeds hosted on GitHub.
Packages downloaded direct from GitHub releases.
I'm not sure what you mean by subscription feed.
The download URL should be whatever the developer wants, but GitHub release hosting is usually the best choice if you host the source on GitHub.
One of the reasons it is hard to "properly package their builds" is that we don't all have builds systems setup for all operating systems. The various operating system builds are basically crowd sourced. So when somebody sends a dev a zip of the build from another OS it gets added to the release. Please keep this in mind. It is not because we don't know how to merge zip files.
@jeremywen I was referring to using .rar files or zip bombs instead of running make dist. Or, using zipmerge from the ziputils package, which merges zip files incorrectly, resulting in files with broken permissions which require sudo rm to remove or messing with chattr. Since zipmerge is broken, we need a script that simply unzips a list of files into a temporary directory, overwriting as necessary, and then zipping them from scratch into a filename given by the last argument.
Ok, I misunderstood then. However, I think my comment is still something to consider.
We could also think of something like Travis CI for automatic building on all platforms since the software is open source? this would not work for commercial closed source plugins though
@jeremywen Sure, I suppose having a single download URL will also encourage developers to collect contributed builds into a merged ZIP file as well.
@sanderbaan Feel free to make a travisci script that does this. Woulld be interested in the results. I looked into it months ago when Rack was a bit harder to build and the task looked too daunting, but now the build system is mostly self-contained on all platforms.
And yes, you're right. It's very expensive at $60/month.
Enzo has some experience with this maybe he has a easy way ;)
@AndrewBelt This will also encourage me to setup a linux environment to build/test in 馃槃
Travis-ci is free for open source projects. I use it to test builds for my plugins on linux and osx, the only things missing is a script to upload the build output somewhere.
Unfortunately, Windows is not supported by Travis-ci. I tried to use Appveyor without success (I did not insist too much). You can see the .travis.yml here (or use it also if someone want, just change the Simple repository url by your own): https://github.com/IohannRabeson/VCVRack-Simple/blob/master/.travis.yml.
Slight (devil's advocate) tangent - what mechanism (if any) would be in place to try to weed out malicious or otherwise non functioning code or would it just be a 'use at your own risk' free for all?
I recall the recent showstopper Andrew just prior to 0.5.0. If we had not alerted the other dev then his package could/would have crashed Rack.
If a user batch installed a bunch of binaries, rather than one at a time, they would have no idea which plugin had caused a problem (if there was one).
You might say that there is little difference to the situation as it stands in this regard .... until you start getting showered with complaints and support requests that is ....
Anyway, I am sure all this has crossed your mind already. Please move this comment if this would be
better discussed elsewhere.
weed out malicious
I imagine with 3-4 maintainers of the plugin manifest repo, it would take no more than 30 minutes to discover and remove malicious plugins that were accidentally added or their download files changed.
otherwise non functioning code ...
It would be unlikely for an accident to cause Rack to crash upon loading the plugin. Creation of plugins would be handled with https://github.com/VCVRack/Rack/issues/467
I recall the recent showstopper Andrew just prior to 0.5.0. If we had not alerted the other dev then his package could/would have crashed Rack.
Link?
I knew you would have thought it through!
Unfortunately, Windows is not supported by Travis-ci. I tried to use Appveyor without success (I did not insist too much). You can see the .travis.yml here (or use it also if someone want, just change the Simple repository url by your own): https://github.com/IohannRabeson/VCVRack-Simple/blob/master/.travis.yml.
It's my understanding that you can cross-compile for Windows/MinGW toolchain using some scripts for Travis under Linux. Basically install the mingw packages and runtime that includes the Windows header files.
I've never done this myself however. There are a number of .yml scripts and discussion that do so in and about the net.
@AndrewBelt
How does a third party developer publish an updated release?
And what is "synchronized copy of the community repo"?
Also how would a developer register/apply?
I am just looking at all this from my (3rd party developer) perspective.
Edit: Removed questions.
Good questions.
Ok this is all making more sense now. More like a way better version of that dumb script I wrote when there was only a few third party modules.
I am willing to help maintain that repo. Or wherever you need me to help ...maybe 馃槃
A similar project is Homebrew, where all package manifests are dumped into one folder. This will do the same. https://github.com/Homebrew/homebrew-core/tree/master/Formula
Yes, I have used homebrew. I have just never implemented something like it before. This is so much more clear now. Thank You.
You hinted at another question: What I meant by "synchronized" repo is that upon every commit (either with Github webhooks or with 5-minute interval polling), the VCV server will pull the latest commit, so that the VCV API handling the plugin management is always up to date.
Ok so I would:
VCV server would:
A few minutes later the user would:
Edit: Added pull request.
Ok, sorry for all my dumb questions! Although, maybe they will also help others reading this.
Yes, exactly, with thw additional step that a repo maintainer would need to accept your PR, if you aren't already.
Would that person (maybe me) only verify the meta data? Would there be any other verifications done?
Ideally someone should download the ZIP in the "download" property manually and check that it doesn't crash Rack and passes through virustotal.com, but if they're a reputable plugin developer with good previous releases, there'd no reason to check it on every version update.
Makes sense. So it really wouldn't take that long to get the build from dev to user. And will save so much time in the long run with people not understanding where or how to get these modules.
Well sign me up. Let's go! I want to be apart of this if you couldn't tell 馃槃
Hooray! Glad to see this is happening, I think this is going to be huuuuugely productive for the VCV ecosystem!
I think the homebrew model for distribution/management is pretty much exactly what I had in mind. The difference there being that brew is a source-based distribution, and this will be a binary-based distribution. This also greatly saves on hosting costs, since GitHub will be the host for all of the compiled community racks.
I mentioned my concerns for the primary use case in this #454 - the primary one being that it should be possible to open .vcv files and (if possible), install the required modules at their required versions. This makes sharing racks _much_, _much_ easier than having to manually search the internet for the necessary modules.
Some questions about the manifest format:
Does version refer to the version of _this_ plugin, or of the _version of Rack_ required? I think that both should be fields in the manifest. I think version should be the plugin and rack_version should be for the _minimum required_ Rack version.
Are we accepting binary module submissions, or should all community modules be built from source and packaged by Rack Community Team? I think that I vote for the latter.
It looks like you're proposing that all architectures download a zip with all of the different versions in it. I think it would make more sense to do something like:
"downloads": {
"linux": "https://github.com/VCVRack/Rack-Modules/raw/master/dist/AudibleInstruments-0.5.0-linux.zip",
"osx": "https://github.com/VCVRack/Rack-Modules/raw/master/dist/AudibleInstruments-0.5.0-osx.zip",
"win": "https://github.com/VCVRack/Rack-Modules/raw/master/dist/AudibleInstruments-0.5.0-win.zip",
}
What should be the terminology for the Rack Community Modules?
It's fine to have the local version of Rack use the web server for managing installs, but the manifest format should be versatile enough that if we create a vpm CLI tool, it will be able to understand that vpm install RJModules will use the "official" API/community repo and vpm install --url https://github.com/Miserlou/RJModules (or similar) will use the one from my personal repo. This will make dev collaboration much easier.
Anyway, I'm sure I'll have more feedback as we move forward but that's just my initial impression. I also suggest that we start a different ticket or repo to discuss setting up the CI/build system.
And of course, I'm more than happy to help maintain the packages and to work on the tooling around package management.
So stoked, this is going to be freaking awesome!
since GitHub will be the host for all of the compiled community racks
Not all, but most. You could put https://example.com/MyPlugin.zip in the download link and Rack would download it just like the others. #123 is a dependency of this issue.
- Does
versionrefer to the version of _this_ plugin, or of the _version of Rack_ required? I think that both should be fields in the manifest. I thinkversionshould be the plugin andrack_versionshould be for the _minimum required_ Rack version.
See https://github.com/VCVRack/Rack/issues/266. I am very surprised that every developer is actually following this standard, as you can see at https://github.com/VCVRack/Rack/wiki/List-of-plugins. (Although I think some have mistaken the version column for the compatibility version, but close enough.)
- Are we accepting binary module submissions, or should all community modules be built from source and packaged by Rack Community Team? I think that I vote for the latter.
No, building should be up to the package distributor, unless a different team than the manifest repo maintainers decides to offer the service of providing builds on a single monolithic Github repo or something. But this has disadvantages: slower release time, developer is no longer in charge of his own binaries, transfer of trust, etc.
- It looks like you're proposing that all architectures download a zip with all of the different versions in it. I think it would make more sense to do something like:
Perhaps. Advantages: easy to replace a single distribution without messing with the others. Disadvantages: Potentially three times longer to move files around, upload to GitHub releases, copy paste URLs. Merging zips is as simple as ./zipmerge.sh, once someone writes a working script.
- What should be the terminology for the Rack Community Modules?
What do you mean? Are you talking about the name of the repo?
- It's fine to have the local version of Rack use the web server for managing installs, but the manifest format should be versatile enough that if we create a
vpmCLI tool, it will be able to understand thatvpm install RJModuleswill use the "official" API/community repo andvpm install Miserlou/RJModules(or similar) will use the one from my personal repo. This will make dev collaboration much easier.
I guess, but you're on your own. I'll be officially supporting the VCV API from Rack using your VCV account, which pulls its information from the community manifest repo.
And of course, I'm more than happy to help maintain the packages and to work on the tooling around package management.
There should be no additional tooling around package management after I write my internal software on api.vcvrack.com. Rack would have to be patched to add OpenSSL support, but that's about it.
Just read the thread and having a "Built by the Rack Team" for me works great. As a user knowing that a binary i'm running is guaranteed to be built from the source I can read is great - much less chance of something sneaky creeping in. Could these kind of builds be marked with an icon?
As a developer I'm used to continuous build integration. A a Git tag of a specific form that would the trigger the build and release on all supported platforms would be awesome! Maybe the release should not published until it has then been tested and ok'd.
I really appreciate that the above is a lot to setup and maintain, but is can be so worth it. I'm happy to help!
Since I'm impatient and I want this to move along, I'll start the repo in a few hours and migrate everything from the wiki plugin list.
watch out for @Miserlou 's plus sign in the version 馃槃 LOL
@gratrix Start a new "Plugin Build Team" thread? I'm fine with it as long as it's the developer's choice to use the team's builds in their plugin's "download" URL.
@gratrix [Posting here as I am out for the day now] As I am already building everything from source you can sign me up as a Mac builder if you like.
After implementing this in full, there is no longer a reason to make Rack's "local" folders visible to the user, so I will switch to hidden directories in #265
Last call for JSON suggestions, creating community repo in 15 minutes.
@Miserlou @jeremywen I've invited you to the repository. Maintaining is simple, just read the README and accept PRs which follow them. Suggest fixes to PR submitters otherwise.
As a bonus, it would be extremely helpful if you could edit some of the JSON files that are already present. I just scraped data off of the wiki list, but most of the download URLs are not direct links to the ZIP files, and the slugs may be incorrect.
ok thanks I will go through them tonight after work and see what I can correct. pfft work...
@AndrewBelt I am sure you have already thought this through but here is my question. How will users select or opt-in opt-out of certain modules? What if they don't want some of the subpar modules or they think JW-Modules suck? Will there be some popup or dialog or something? Or is this a non-issue, and they get whatever is available?
They will be able to add and remove them from the future vcvrack.com/manager page. All modules are not added to your account by default, except Fundamental, which comes with Rack.
Perfect! Will there be screenshots on that page like the current ones or something?
It'll be a pretty dry design, similar to the old wiki list except with Add buttons and a login form. It is not meant for marketing.
Ok maybe in the future there could be a json entry for a certain size image or something. That would be helpful to users.
btw guys, i might have missed this but when is this going to be working in the vcv software? is that going to be in a next update?
My plan is that it'll be added to Rack v0.5.1.
Just wanted to throw in my .02usd and say that in my limited experience, creating your own package manager backend is generally more trouble than it's worth.
Creating a package manager from an initial set of minimal requirements seems like a straightforward and tractable problem, but it has a lot of subtleties and can, surprisingly quickly, become a time sink and source of tech debt. What's worse, the requirements from one package manager use case to another aren't that different, so there's not a huuuge payoff to having rolled your own.
It might be a good idea to evaluate whether an existing stack could serve VCVRack's needs, maybe not now, but in the future if the backend requirements start to grow. Off the top of my head, both Nuget and Conan support C++ packages and have open-source, self-deployable server implementations. (Nuget is generally pretty good, I don't have any direct experience with Conan.)
I don't understand your point. It'll be 60 lines of Node/Express and take an hour and 30 minutes to write, since much of the code already exists in both Rack (the client) and api.vcvrack.com (the server).
I'm just waiting on some free time, and the VCV-community repo to be seeded with initial manifests.
I completely agree with @AndrewBelt Also, in my experience integrating some else's overly complex system into yours cause cause even more trouble than it's worth. I don't know how many libraries I have used and ended up regretting it because I depended on it so heavily.
I'm just saying that I have been down this road before, and at the time, we said the same things you're saying now, and we later regretted it. :)
I'm probably being overly paranoid. If you never do anything more complicated than "install the latest version of this self-contained plugin package" then you should be fine.
But if at some point you want to support things like a patch repository, where you can pull down a sample patch and all of the plugins that the patch uses, or keep multiple versions of plugins used by different packages on the local hard drive and load the versions on-demand, then using someone else's already built and tested dependency resolution infrastructure could start to look attractive.
I'm probably being overly paranoid. If you never do anything more complicated than "install the latest version of this self-contained plugin package" then you should be fine.
Then I'm fine.
But if at some point you want to support things like a patch repository
God, no
multiple versions of plugins
Ahhhhh, stop
Hi. Very interesting thread for me since I raised idea of collecting user created script for REAPER 2 years ago. There were previously scattered through different web places. Today there is ReaPack extension. It is a package manager for all (supported) kind of user-created stuff. A lot of its good features can be usefull for VCVRack also.
I donn know what will be with backward compatibility, seems youll need to mark installed plugins as grey if their version lower than current VCVRack version (so they willn`t be uninstalled i.e. awaiting update from author and unavailable)
Sorry if i sounds unclear (not English speaker at all).
Michael.
there is dedicated common repository for relatively simple user created stuff (we did it like this ), maintained by some users, for VCV it could be a pack for simple modules/utilities easy to learn (well commented),
I think the Fundamental plugin serves this purpose well. If you start with 0 modular knowledge, it would take weeks to actually reach the full patching potential of those modules (and even I'm learning new things to do with them). After that, you have a pretty good idea of what you want and can branch out into other directions of your choice (drums, polyphony, physical modeling, Eurorack ports, etc).
some big single user repositories bundled into manager as trusted (potentilly Autodafe, Hetrick)
I believe everything should go into one repository and listed as a flat list (sortable alphabetically, last updated, popularity*, etc), and every module that remotely works at all should be listed under it.
The package manager will be integrated with the users' accounts, so Rack only needs to ask "what plugins are available, and what are their versions". Then it compares the versions of the currently loaded plugins. Very simple.
there is an about page for every repo where user can see where to put bug reports, feature requests and donation to plugin/script author
That would just be plugin developers' github repos.
there is a possibility to revert back to previous version of plugin (like if newer version has bug)
Precisely. There is no version less-than comparison anywhere. Simply string comparison, so if MyPlugin v0.5.3 has a bug, the community repo will be "updated" to MyPlugin v0.5.2 and Rack will download that version again.
I donn know what will be with backward compatibility, seems youll need to mark installed plugins as grey if their version lower than current VCVRack version (so they willn`t be uninstalled i.e. awaiting update from author and unavailable)
Incompatible plugins can't loaded by Rack, so Rack will consider them as nonexistent. So when it compares existing plugins with the list, it will think it doesn't have them and download the new versions.
*I haven't thought about this, but sorting by popularity will be a great idea, and since it's integrated with VCV accounts, it's easy for it to know how many have added it to their account.
馃憤 on the popularity idea. I have always wanted to be popular. Maybe, just maybe this will be my chance. LOL 馃槃
(Yes, not the favorite for maybe 0.2% of users, but they always have the option of manually downloading the plugins from the download URLs and placing them in the plugins/ directory.)
It's fine to have the local version of Rack use the web server for managing installs, but the manifest format should be versatile enough that if we create a vpm CLI tool, it will be able to understand that vpm install RJModules will use the "official" API/community repo and vpm install Miserlou/RJModules (or similar) will use the one from my personal repo. This will make dev collaboration much easier.
Maybe the main rack binary could provide simple flags/commands to synchronize an external manifest URL (fetch, download version, add to local database) and remove a package?
$ ./Rack --install https://example.com/Module.json
$ ./Rack --remove https://example.com/Module.json
$ ./Rack --remove ManufacturerSlug/ModuleName # or perhaps this?
those 'manual install' packages could be tracked similarily to the ones synchronized from the online user API and live in a 'Local/Other' category, so that changing your subscription status online will either move re-classify that plugin as 'Local' or uninstall it. Local plugins probably shouldn't autoupdate which has the benefit of potentially making development simpler.
Maybe local plugins like this wouldn't even need tracking in VCVRack and would just be discovered as the plugins found in the directory that are _not_ tracked by the API.
Since it's 0.2% (arbitrary number), it's a very low priority.
But it seems possible at first glance. It would still use the api.vcvrack.com server to query packages.
Removing individual modules is impossible. The manager will operating only on Plugin Slugs (there is no such thing as a manufacturer slug, just a manufacturer name).
Something like
./Rack add MyPlugin
./Rack remove MyPlugin
About the travis-ci things, I just successfully cross-compiled a plugin to windows on Arch Linux, you can see my changes and tiny log in #509. I think this should be reproduceable in travis, especially if the Makefile can have an extra target that doesn't build the core modules.
@AndrewBelt right, I lost my train of thought there a bit, Module.json should be a plugin by one manufacturer and removing single module doesn't make sense.
As a personal preference, I'd recommend to use something existing instead of reinventing the wheel.
I guess NuGet might be a good option, since it is well established on all plaforms (Win/Mac/Linux) and much more stable than NPM and most other package managers.
Not sure if you can use if for not-.NET workloads, but would recommend to port to .NET Standard/Native anyway, because it is independent of the execution platform and also independent of the programming language.
@MovGP0 As mentioned before, this isn't something you can just drop a premade solution. All the Rack Package Manager does is 1) queries the vcvrack.com API with the user's token, 2) checks the installed plugin and compares them with available plugins in the user's account, and 3) downloads, checks SHA256, and unzips them.
This is not a personal preference thing, it is a problem that has a clear solution, and it is not NuGet, npm, etc.
Most helpful comment
Then I'm fine.
God, no
Ahhhhh, stop