360controller: Notarization of kexts (including the one used by 360controller) will be required by MacOS 10.14.5 and later

[PokeCaptain]

Type of Controller

All

OS Version

10.14.5 β2 (18F108f)

Driver Version

0.16.11

Connection Method

Wired (Applies to all)

Device Name and Info

N/A

Response of Controller

Logo lights up (usb power), no recognition by driver (as it's not actually loaded).

Response in Preference Pane

Not detected

Detailed Explaination of Issue

As added in the Developer Beta 2 of 10.14.5, notarization of the kext and its installation package will be required to install it. Non-notarized kexts will be blocked. This is now documented on Apple's developer documentation page here. Unfortunately, the notarization that this driver would be forced to go through requires Xcode 10, which is a problem documented in previous issues.

That leaves two options:

• Rewrite this driver and package with new frameworks, allowing it to be developed with Xcode 10 and continuing to allow new installations or

-Discontinue the driver entirely

Neither option is ideal, and neither will be an easy decision. I personally hope this driver is maintained, but I understand if it is too much effort to rewrite it.

[FranticRain]

A rewrite of the driver is currently impossible* because of a memory leak in the modern version of the IOKit driver libraries. You can find some more discussion about this in #809

*when I say it is impossible, it's possible. It would just mean that the delivered product would induce a memory leak on all user's systems. Also, development would require a reboot for every debug build. So while it is possible, it's excruciatingly arduous, and I have no interest in pursuing it.

[Coder-256]

@PokeCaptain I'm happy to say that you're wrong! To use notarization, Apple says you need the 10.9 SDK or later, but I'm pretty sure that Xcode 6 luckily does use the 10.9 SDK. Furthermore, they explicitly say:

Important

You can notarize an app that you build with earlier versions of Xcode, but you must use Xcode 10 or later to actually perform the notarization. This is because the altool utility, which you use to perform notarization, only supports notarization starting in Xcode 10. See Customizing the Notarization Workflow.

For example (untested):

SDKROOT=/Applications/Xcode10.app/Contents/Developer xcrun altool --notarize-app # ...

Even if your xcode-select is Xcode 6, since you can override the SDKROOT, I think that should actually work since we're in the sweet spot of the 10.9 SDK: Old enough to use the old USB libraries, but new enough to support notarization. Apple actually includes a fairly comprehensive guide on the workflow for how to notarize through the command line. Fair warning: I haven't tested this since I don't have a developer membership.

@FranticRain This is fixable (most likely), and I agree with @PokeCaptain that it is very important to fix ASAP before 10.14.5 becomes stable and peoples' controllers stop working. In fact I'm pretty sure you would only need to modify your distribution workflow to include altool (and wait for notarization to complete, which will definitely not be fun), but no actual code changes should even be necessary really.

[FranticRain]

I built the latest version of the driver and submitted it to be notarized. I'll let you know when the process finishes up. Will need to get someone to try it in 10.14.5 beta to make sure that it actually works. Looks like it shouldn't be too hard to integrate into the makedmg.sh step of the build, assuming this works.

@Coder-256 Thanks for the heads up an information. I was losing my mind scrambling to try and update the driver to the new version after suddenly realizing that the memory leak that I had been complaining about had been fixed. But now I can take my time and not destroy my life for FOSS.

[Coder-256]

@FranticRain Do you mean that radar 40552414 has been resolved? And that #809 can be fixed? That's great news!!! Now I wish I hadn't helped, since wireless support will be pushed back haha ;) Anyway glad to hear that, and it's great news that this project will finally be able to use the latest SDK again after all this time. Can you push the branch you're working on? I'd love to see how it's going (and maybe help out a bit).

Also I'm glad that notarization seems to be working so hopefully the driver won't be blocked regardless of the status of the fixes (or is it a rewrite?). But something that occurred to me is that you might run into issues with hardened runtime... hopefully notarization will be approved but if not let me know and I'll look into how to fix that for older SDKs

[FranticRain]

@Coder-256 As far as I can tell, yes. That has been resolved. At least based on my cursory testing. And yes, that hopefully means that we can get wireless controllers working again. I'm just messing around with some of the ideas I've had for a rewrite at the moment, because it solves some of the issues we had with the last time we did a rewrite. Not sure if changing the layout of the driver will completely ruin support for the driver, though. Lots of games hook into the driver in REALLY dumb ways. So we'll see how it goes. But the work I've done can be seen here: https://github.com/FranticRain/Xbox360ControllerRewrite

The hardened stuff did cause my first attempt to fail, but the documentation was really good and told me how to fix it easily. It's so refreshing to have actually good documentation after spending all of this time in the depths of kernel land.

So here's a signed and notarized version with a stapled notarization, so maybe this will work? Thanks for making me have to put it in a zip, Github. 360ControllerInstall_0.16.11_Notarized.dmg.zip

[Coder-256]

Exciting! I don't have the beta so I can't confirm... but on 10.14.4, stapler validate -v 360ControllerInstall_0.16.11_Notarized.dmg works! That's a good sign

[Coder-256]

Hmm... This doesn't look too good... are you sure you notarized everything and not just the dmg? I'm not 100% sure if that's necessary or not, but I'm pretty sure that it is necessary to notarize at least the kext if not also the pkg. Here is the output for the installer pkg and the kext I extracted from it:

$ stapler validate -v /Volumes/360ControllerInstall/Install360Controller.pkg 
Processing: /Volumes/360ControllerInstall/Install360Controller.pkg
Properties are {
    NSURLIsDirectoryKey = 0;
    NSURLIsPackageKey = 0;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Installer package";
    NSURLTypeIdentifierKey = "com.apple.installer-package-archive";
    "_NSURLIsApplicationKey" = 0;
}
Sig Type is RSA. Length is 3
Could not find an appropriate "code signature" in the Install360Controller.pkg installer package.
Cannot download ticket. CDHash must be set.
$ echo $?
66
$ stapler validate -v 360Controller.kext 
Processing: 360Controller.kext
Properties are {
    NSURLIsDirectoryKey = 1;
    NSURLIsPackageKey = 1;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Kernel Extension";
    NSURLTypeIdentifierKey = "dyn.ah62d4qmuhk2x445ftb4a";
    "_NSURLIsApplicationKey" = 0;
}
Props are {
    cdhash = <f1d9eaef d7e91d07 4d8216e2 2028b175 e67569f0>;
    digestAlgorithm = 2;
    flags = 65536;
    secureTimestamp = "2019-04-17 02:43:12 +0000";
    signingId = "com.mice.driver.Xbox360Controller";
    teamId = G947KKM5RL;
}
360Controller.kext does not have a ticket stapled to it.
$ echo $?
65

What script did you use to run altool?

EDIT: Nevermind, according to Apple this should still work, my mistake:

Alternatively, you can put apps, kernel extensions, and other software in a container, like a disk image, and notarize the container. The notary service accepts disk images (UDIF format), signed flat installer packages, and ZIP archives. It processes nested containers as well, like packages inside a disk image.

Anyway I guess the real test would be confirming with somebody who has the beta.

[FranticRain]

Yeah. I read that same line so I only ran the command against the DMG, but we will have to wait and see I guess. I'd rather not risk it with all the work that still needs to be done on the driver, but if we get closer and it hasn't been tested, I'll jump on the beta.

[PokeCaptain]

WOW! This is the first time in a long while that I’m actually happy that I was wrong! The fact that the radar was finally fixed is even better news! I have the latest beta myself and will test the notarized version in the morning. In the meantime, CONGRATS!

[Coder-256]

@FranticRain I definitely wouldn't install the beta yet! (although I guess you could install it on another partition). Although I guess if you disable SIP then you should still be able to test ont beta if things end up not working. I'm just not really sure how easy it is to downgrade if you need to (you'd probably need to restore from time machine)

On a separate note, I took a look at what you've started working on, and although it's not much, I see that it is being designed as a kext. I can't help but wonder, does it really still need to be a kext? I think that one major purpose of the IOUSBFamily redesign was that now you can it is much easier to access USB devices 100% in userspace with IOUSBLib.h. Especially for something as high-level as a Xbox controller, I think that this is almost definitely possible. Moving to userspace has a ton of benefits: no more kernel panics, doesn't need admin privileges to install/update/delete, more secure, is lightweight/easier to distribute, possibly even MAS (yes seriously), etc.

The only problem is that you probably can't create a virtual HID device from outside the kernel. But I think you can definitely access controllers from outside the kernel by interfacing with them as generic USB devices, just you can't necessarily create drivers for those devices to act as HID devices. Then the only challenge is sending events to games in a way that they expect; that might only be possible on a case-by-case basis, I'm not totally sure. An analogy is if you have some kind of custom USB hub, you can probably read from it in userspace, but other apps (and the system) might not be able to talk to the connected devices with the system USB APIs unless you provide a driver; and due to the way IOKit works, drivers have to be in the kernel for no good reason.

I guess what I'm saying is that if you can somehow talk to games without using an IOKit driver (eg. if you can simulate a controller without a kext), then you probably don't need to use a kext since you can read/write from USB and Bluetooth without one. But that might just not be feasible (maybe games just wouldn't be compatible), and I guess there's nothing that anybody but Apple can do so a kext would be justified. (99% of kexts could definitely be safely outside of the kernel if only Apple would allow it </rant>)

[FranticRain]

Plain and simple, some of the ways people use the driver are incredibly stupid. There are some games that require the driver, but grab the data packet from the controller before it has been fiddled with, making Xbox One and Xbox 360 controllers have different button mappings. (_Here's looking at you, Awesomenauts_) For reasons like this, I have 0 faith that a user-space rewrite will maintain any level of backwards compatibility. And at the end of the day, people who use this driver don't give a crap about safe vs. not. They just want it to plug in to their system, work, and work with the games they use it with. Any level of regression would just cause a level of issue headache that I'm not interested in dealing with.

There are even some games that look for com.mice.driver.Xbox360Controller specifically, so if you change the organization, but have the exact same code, it won't work. Heck, I can't even fix minor issues with the HID descriptor because it will break compatibility with some games. macOS support for non-MFi controllers is atrocious, and individual game support is even worse. At the end of the day, we have to make a choice for 360Controller. Is it the most compatible driver for Xbox controllers? Or is it the safest driver for Xbox controllers, compatibility be damned?

I'll give a user space driver a shot when I get a chance, and maybe all of my concerns will be completely unfounded. But I don't know. This driver is a consistent thorn in my side, and I get nothing out of supporting it. I don't game on macOS or anything like that, so I don't even eat my own dog food. If user space means that this driver doesn't require me around to support it anymore, then it may be the preferable option, compatibility be damned.

[Coder-256]

Sadly I think that userspace would break too many things to be worth it now that I think about it more but it's worth trying

[PokeCaptain]

Just tested, can confirm the driver works again with notarization. Crisis averted.

[FranticRain]

Excellent news. I'll update the release with the notary stapled version when I get a chance and close out the issue at that time.

[electric-monk]

The reason I made it a kext in the first place is that the "documented official way to implement a game" on MacOS is to use the HID library. At the root level, the XBox 360 controller is a fully compliant HID device, just missing one USB HID descriptor to make it work everywhere out the box. I did actually use all the testing of the functionality using OSX's userspace USB stuff, because obviously that made it so much easier to figure things out like the rumble motors.

I specifically chose to make it a kext to support games that _already existed when I started implementing it_. There was no point in implementing a userspace driver at that point since that would have not done what I set out to do with it - playing existing games with an XBox 360 controller.

At the very root of it, the driver was only a few lines long, behaving as an ordinary HID device in all but "get HID descriptor", which the driver injected one I'd made myself for, thus making OSX + games happy. Anything added on top of that (dead zones, etc.) was user-requested features on top. I note the rewrite looks almost exactly like the original form of this driver.

It was a very unexpected side effect that games would start looking for my bundle ID, and even more surprisingly, my actual C++ class names, which caused me to have to jump through hoops to refactor/add features even when I had time to do so (and I appreciate the huge piles of work others have been doing on this branch).

I do apologise for the wireless bit though - I had imagined the other 360 wireless controllers would expose different interfaces, so had a go at making it a "bus" type driver where such child drivers could attach. That was all probably a bit pointless in the end, though it should also be possible to pull it all back into one kext and having it not be so "exciting" whilst also not upsetting any games looking for the classnames (one of my previous plans was to refactor slightly so the actual bottom level '360 controller' class could attach to either the wired or wireless devices, which would have improved things).

[FranticRain]

@electric-monk Thanks for the insight. I definitely respect the commitment to compatibility, and that's definitely something that I'd like to keep up. Even if it isn't the way that Apple wants us to use the operating system.

I'll see if a rewrite attempt just completely tanks on certain games. If it does, we can always fall back to the old style. But my main hope was to shift the driver to match on vendors and interfaces instead of individual PID/VID combos so that we could better match devices without needing a new issue for every color variant that PDP et al. decide to put out. But if it ruins compatibility, I'll just have to live with it the way it is, and how incredibly obnoxious that is going to be with notarization.