Pmbootstrap: Make proprietary firmware and userspace optional

[ollieparanoid]

The built-in wifi and baseband will not work without proprietary firmware on practically all phones. However, we could allow the user to avoid installing these blobs, just like libre distributions do it.

Implementation idea

1. Change the device package recipes to generate the following subpackages:
2. device-vendor-name-nonfree-userland (we don't have proprietary userland right now, so this is not needed)
3. device-vendor-name-nonfree-firmware (depends on the firmware package(s))

(This is really easy in APKBUILDs and can be done with very few lines.)

1. Ask in pmbootstrap init:

(Explanation text)
(pkgdesc from the nonfree packages, so they can tell which hardware components they enable)
Use non-free components (none, firmware, userland, all)? [firmware]: 

We could directly skip the question, in case there is no nonfree subpackage for the device, and only show the firmware/userland choice when such a package exists.

1. Depending on what the user chose, the nonfree packages and its dependencies get built and installed in pmbootstrap install.

Advantages

• Make people aware, that there are proprietary components and what they are for
• People who prefer a full libre stack can disable the firmware files which are enabled by default right now (they could use an external modem via usb etc.)
• It would allow packaging proprietary userspace drivers with libhybris (#678) in theory, while keeping it cleanly separated for users who do not wish to install them.
• Once we can run proprietary components in pmOS, it might be a lot easier to reverse engineer them.

Thoughts?

[drebrez]

The idea to let the user choose during the pmbootstrap init looks good.

What is not really clear, at least for me, are the options of the question.
none and all are fine, but what exactly means firmware and userland?
With firmware I suppose proprietary blobs in the kernel, right?
Choosing firmware means also having proprietary blobs in userland?
Use something like kernel-land only and user-land only makes it more clear?

[ollieparanoid]

Firmware

With firmware I mean blobs, that other chips on the phone require to function. Such as the Wifi chip. They are not really part of the kernel, but the kernel loads them from the disk and passes them to these chips.

Speaking in package names, this would be any firmware- package from postmarketOS, and the linux-firmware package provided by the Kernel developers (they only collect the firmwares, the binaries come from the vendors).

Userland

With userland I mean blobs running in userland, such as proprietary Android RIL implementations (#598) or the userland GPU drivers (which work in combination with open source kernel module stubs, just like the proprietary nvidia driver for PCs as I understand it).

(In a broader sense, when we would port Anbox or similar software to run Android programs, WhatsApp etc. would also be proprietary userland programs. But differentiating between them and Free and open source programs from F-Droid running in Anbox isn't that easy and not worth discussing about right now in my opinion.)

[ollieparanoid]

we don't have proprietary userland right now, so this is not needed

This is going to change soon, as the userland blobs for talking to the GPU of the Nokia N9 have been packaged for postmarketOS. This is not in our master branch yet, so let's figure out how we want to have it.

My proposal:

• Rename aports/firmware to aports/nonfree-firmware
• Create a new folder aports/nonfree-userland
  
  
  
  • put the N9 userland blob packaging in there
  
  
• Implementing the question for proprietary components in pmbootstrap init as outlined above

I have also considered naming the new folder blob, but in the second naming scheme I came up with the nonfree- prefix makes it clearer that blob and firmware are both not FLOSS, and that this is the reason why we put them in extra folders at all.

What do you think, everyone?

[PureTryOut]

Sounds good. I'd go with the nonfree- naming scheme personally, just is a bit more obvious what it is.

[ollieparanoid]

TODO:

• [x] Get #1254 merged (pmbootstrap changes)
• [x] Document how one can use the feature in the device specific packages wiki page
• [x] Change all device packages, that have firmware packages to use this scheme (#1268)
• [ ] Rename aports/firmware to aports/nonfree-firmware (#1274)

[MayeulC]

I am a bit uncomfortable with having non-free blobs in the master branch. Also, do we have the rights to redistribute firmwares? (#797-related, thanks @ata2001)

I would be much more comfortable if it was in another repo altogether. Maybe we could add an "overlay" feature that supersedes pmbootstrap's packages if they are in some specially-named folders? That way, multiple projects could live on top of pmbootstrap, just a git clone away.

There are a few reasons to want to do this:

• Firstly, it allows us to concentrate on mainlining devices in the main tree (I saw in #1039 that mainline wasn't used because the constructor fork would work better with libhybris). Having two configurations would solve this problem.
• It's generally a good idea to make proprietary software live in another repo
• This could be something of a separate project, allowing pmos (the base distribution) to earn the "respects your freedom" certification (even if it could be misleading due to baseband firmware, etc...

As for the wiki, I really don't know what's best. I really like the idea of having device information centralized. But maybe add a guideline that everything non-free should be in a dedicated category?

So, these are my two cents. What are your opinions on this?

[ollieparanoid]

Blobs in master?

@MayeulC wrote:

I am a bit uncomfortable with having non-free blobs in the master branch.

This is probably what you've meant, but just to make sure no one gets this wrong: We don't have blobs in the master branch, we have package build recipes (APKBUILDs) which allow to build packages that have such blobs. They can easily be differentiated from the other packages, because they are in an extra folder (aports/firmware, to be renamed to aports/nonfree-firmware) and have a prefix (firmware-).

Right now, firmware packages get automatically built/installed when you build any device package where WiFi works for example. The idea of this issue is to change that, so you can build/install the device packages without building non-free firmware (and possibly in the future: userspace) driver packages.

Having proprietary drivers around at all?

@MayeulC wrote:

I would be much more comfortable if it was in another repo altogether. Maybe we could add an "overlay" feature that supersedes pmbootstrap's packages if they are in some specially-named folders? That way, multiple projects could live on top of pmbootstrap, just a git clone away.

There are a few reasons to want to do this:

• Firstly, it allows us to concentrate on mainlining devices in the main tree (I saw in #1039 that mainline wasn't used because the constructor fork would work better with libhybris). Having two configurations would solve this problem.

Motorola Droid 4

The Droid 4 from the pull request you have mentioned is a special case, because in contrary to almost all other devices on which pmOS boots, there's active mainlining progress and most stuff is working already (recent FOSDEM talk). That means going with mainline for that device is a no-brainer for postmarketOS.

So much for the theory. Practically, @NotKit was the only contributor who had access to the device. And his opinion regarding mainline on the Droid 4 is:

@NotKit wrote:

Ideally I would package both, since mainline support and running newer kernel is cool, but it lacks 3D acceleration, cameras and voice calls, which is possible with Android one with libhybris and proper middleware.

(This weekend the situation changed, there are 3 people now who have a Droid 4 and are interested in porting pmOS to it. They may decide to go with mainline now, or with both, and I will as usually help with making either solution work. pmOS does have support for multiple kernels per device btw, although that isn't really used at this point and may need some tweaking.)

Proprietary user-space drivers and postmarketOS

Personally, I prefer running with mainline and no accelerated graphics over having accelerated graphics with userspace blobs and being stuck with a downstream kernel fork. But NotKit is a skilled developer who, besides his huge libhybris work, has put a lot of work in porting Hildon for postmarketOS already (which is great for the whole project, no matter if one wants to run mainline or not - especially now that it's maintained very actively by Maemo Leste).

The Nokia N9 is a similar story: Filippz is mainlining it, and ported postmarketOS to it. He also made proprietary userspace blobs for the PowerVR GPU working on it (with the mainline kernel, not integrated in the master branch yet).

So I had a conflict of interests there: not supporting blobs vs. helping users accomplish what they want to do with pmOS. And I decided to go with my recommendation of integrating libhybris (userspace blob drivers in general), but making them entirely optional (and while we're at it, make firmware files optional as well). I think that is the way it fits best with what we wrote in the 100 days of postmarketOS post:

The postmarketOS community is a collective group of hackers who contribute to this project in their free time. We won't tell someone who wants to, for example, extend postmarketOS to run Doom on their smartwatch that their idea has no benefit to the project's vision. Such activities demonstrate the flexibility of postmarketOS and oftentimes leads to improvements to the project's codebase as new requirements are implemented to cover previously unforeseen use cases. In addition, these fun activities also increase our collective knowledge about the software and hardware we work with. But most importantly we don't want to, or plan to, take the fun away. Because without being fun and rewarding, a free time project becomes a dead project.

Split it into an extra repository or distro, make one of them FSF approved?

@MayeulC wrote:

• It's generally a good idea to make proprietary software live in another repo
• This could be something of a separate project, allowing pmos (the base distribution) to earn the "respects your freedom" certification (even if it could be misleading due to baseband firmware, etc...

As for the wiki, I really don't know what's best. I really like the idea of having device information centralized. But maybe add a guideline that everything non-free should be in a dedicated category?

You're basically saying: Keep the non-free stuff around for people who want it, but split it from the main distribution so we can get the Free Software Foundation's Respects Your Freedom approval. Reading up on this again, the RYF is for hardware actually, but the GNU FSDG is the software counterpart.

I see a couple of problems with that approach for postmarketOS in general:

• Putting a barrier on installing non-free components (via extra repo, split documentation, telling them they must use FSF language in the documentation, ...) increases the development barrier, which is clearly an anti-goal of postmarketOS.
• Easily switching between free and non-free drivers is great for reverse engineering.
• Creating a derivative distribution requires at least one maintainer.

Besides that I can think of a lot more points that are debatable for FSF endorsement (Alpine not using the GNU stack, postmarketOS being a phone distribution without using cellular modem firmware is a joke, I think binary patching security holes in firmware is a good idea, what about the environmental aspect of not throwing devices away which only require one blob we could sandbox to still use them ...). I am not sure about what the FSF's points of view are on those exactly (PureOS is FSF-endorsed nowadays, does that also count for the Librem 5 and what about their Wifi firmware, do they work around that in hardware, so it can not be updated from the main OS? what about security updates then?). This leads me to another point: I don't have the time or energy to figure that stuff out. And whoever else would try to get the FSF approval for pmOS would have to deal with this as well.

Dude wrap it up

In conclusion, I think the FSF does a lot of great things and I would love to have devices entirely open sourced, including firmware, and I think the hacker community is skilled enough to do this with enough time (years) and manpower. But for the reasons outlined above I don't see why we should split postmarketOS in a libre and non-libre distribution.

Thanks for sharing your thoughts @MayeulC!

[MayeulC]

Thank you for the great writeup and in-depth answer, I generally agree with all the above points, which means I'm OK with the current approach.

Kind of off-topic, but just an idea regarding multiple kernels: maybe pmbootstrap will be able to offer the user a choice at some point; I think the way pacman handles multiple providers is nice, and such a mechanism could be unified for choosing between DE, kernel and other components.

[ollieparanoid]

The only thing left to do would be renaming the firmware folder, but since we don't have nonfree userspace blobs at this point, it leads to pointless confusion right now (see @z3ntu's reasoning in https://github.com/postmarketOS/pmbootstrap/pull/1274#issuecomment-369739406).
So... everything's implemented for now :+1: