Pmbootstrap: libhybris support?

[NotKit]

libhybris is a compatibility layer for loading Android libraries (compiled against Bionic) by glibc-based Linux applications. However, with some changes it can be compiled with musl as well. Caveats yet to be found, but I was able to utilize Android OpenGL libraries on at least two devices (Moto Droid 4 and Moto Z Play).

Does it make sense to package libhybris for postmarketOS, and if yes, what's the best way to do it? Building libhybris requires extracted Android headers, ideally from device's Android tree. SailfishOS compiles libhybris for each device, while UBPorts/Hallium tried to keep single libhybris build for multiple ones, but now provide at least two libhybris packages - normal version and libhybris-caf for Qualcomm devices due to changes in gralloc headers, making it binary incompatible.

Another problem than would be packaging device-specific Android bits. SFOS uses files from CyanogenMod/LineageOS install, but overrides init scripts and libs. Halium instead builds minimal Android rootfs and runs it inside LXC.

[ollieparanoid]

+CC: @bhush9 (bshah), @z3ntu from Halium

First of all, great research!

Proprietary blobs in userspace

My personal opinion, as written in #postmarketOS:

I don't like them and I prefer solutions like freedreno or tinygles. But I think as long as we keep them optional, it's better to allow them to get more people to contribute to the project instead of forbidding them. I'll write something like this down to the wiki, a "guidelines for proprietary blobs" page or something (as initial draft open for discussion as always).

Chroot instead of libhybris?

I'm wondering if it is possible to run ldd on the blob in question, and then copy the blob with all libraries it links against (including bionic libc) to a folder on the device, and run it in a chroot. As I see it now, it would save us from the complicated compiling steps, the libhybris layer, and maybe we wouldn't even need to worry about using a specific kernel. But that would be too easy, so where is the catch?

Hardening

I would like to restrict the blobs access as much as possible. This should be relatively easy with firejail for example (packaged in Alpine). I would chroot it, restrict network access, apply seccomp filters so it can do nothing but be the graphics driver.

Packaging libhybris

If we really go down that path:

• having two versions of libhybris is better than one per device in my opinion, we could package both (in one aport, with a subpackage for libhybris-caf)
• I would prefer not running Android in LXC, but using native libraries/programs like SFOS (whatever that means)

[z3ntu]

I don't really like libhybris either, it kind of taints GNU/Linux and it's not viable in the long run. Open source graphics drivers (and other drivers) are our future!

[NotKit]

@z3ntu I can agree with you about future, but what can we do about less popular current devices, which are not likely to get open source graphics and other drivers support?

@ollieparanoid As for chroot approach and why libhybris is needed in the first place, Linux apps need to interface with proprietary OpenGL libraries for 3D acceleration, so having blobs on their own is not very useful.

[NotKit]

I made packaging for the forked libhybris to work with musl and later plan to package LXC container configs found in Halium: NotKit/pmbootstrap@cf062d5ea0b08c9b99c5ecceeaba84ba0c433996

The bad thing I found out about android-headers is that they are basically Android version and device-specific. For example, Android 7.1-caf (Qualcomm devices) adds field to hwcomposer layer struct, so any software compiled against those headers (libhybris, kwin_wayland and so on) won't work with devices using Android 6.0 or any other base.

In case of having device-specific packages (like android-headers-addison, libhybris-addison, kwin-addison), what would be the best way to avoid APKBUILD code duplication?

[JBBgameich]

I'd do it like in Halium, that there are android-headers for each used version of android (in Halium 5.1 and 7.1) in a caf and a generic variant. As in postmarketOS all packages are built for a device, you could just add android-heades-7.1-caf as a dependency (does Alpine have optional deps or suggestions like debian?) of a device-$vendor-$codename package and so libhybris would be built aginst the correct headers variant (caf / generic + android version, not headers for each device) fitting the device.

Btw, do you plan to submit your musl compatibility patch for libhybris upstream?

[NotKit]

"postmarketOS all packages are built for a device" - this is not going to be the case when binary repo are introduced, or simply when you build for multiple device. They are build for arch instead.

Yes, I'd like to, but first want to make sure it's actually compatible without major issues. Hooks for some functions had to be disabled, so bionic version is used instead. This may or may not backfire later. For example, currently I have test_hwcomposer and xf86-video-hwcomposer working with it, but didn't manage to get qt5-qpa-hwcomposer displaying anything, which might be either due to differences within Qt version/Qt build, or problems with libhybris.

[ollieparanoid]

@NotKit: thanks for explaining why chroots won't work for graphics, it makes sense now!

What I still don't understand is: you wrote, that when compiling kwin against the headers of one Android version, it would not work with any other Android version.
But we use shared libraries, right? So I thought you compille kwin against the regular libGL.so, and then you can swap that out with libhybris if you like to, and the whole purpose of this library is to be a drop-in replacement. What am I missing?
Providing a version of every program, that links against GL for every Android version doesn't sound good at all (it would be in addition to the "proper" version, that works without libhybris anyway).

@JBBgameich: thanks for the insights on how it works in Halium, and having one Android headers package per Android version does sound much saner than having them for each device.

@NotKit is right, the whole point of the project is, that we do not build everything for each device, but that we can share packages between all devices of the same architecture (armhf, aarch64, x86_64), just like it works in traditional desktop Linux distributions.

But maybe we could have one binary libhybris package per Android version, would that work?

does Alpine have optional deps or suggestions like debian?

Alpine has install_if, which works like a reverse optional dependency/suggestion/recommends. What we could do in theory for devices supporting libhybris:

pkgname="device-some-example"
subpackages="$pkgname-hybris"
# ...
hybris() {
    install_if="$pkgname libhybris"
    depends="libhybris-8.0"
    mkdir "$subpkgdir"
}

We would have an (empty?) libhybris package, and when that gets installed, and you also have that device package installed, it would automatically pull in libhybris-8.0, which I just made up to contain Android 8.0 specific libhybris stuff.

(Could someone quickly explain what caf and generic is?)

[JBBgameich]

Caf and generic is only a different configuration of android-config.h in the headers. Caf is normally for Qualcomms Android and generic for AOSP.

[z3ntu]

CAF = Code Aurora Forum

https://www.codeaurora.org/project/android-for-msm
https://source.codeaurora.org/quic/la

[NotKit]

@ollieparanoid

But we use shared libraries, right? So I thought you compille kwin against the regular libGL.so, and then you can swap that out with libhybris if you like to, and the whole purpose of this library is to be a drop-in replacement. What am I missing?

It works the way you expect for libEGL/libGLESv2 (not libGL though, since it doesn't provide GLX/desktop OpenGL). libhybris provides abstraction for Wayland apps.

The problem is that packages like qt5-qpa-hwcomposer (used by Nemo/LuneOS), kwin and so on also use HWComposer API to get output on device screen. libhardware (wrapped by libhybris) is used to access device-specific HWComposer module (like in https://github.com/libhybris/libhybris/blob/master/hybris/tests/test_hwcomposer.cpp#L181). So that apps have to include HWComposer headers coming from specific Android version/SoC vendor, which provides structs which are then feed to HWComposer module.

Changes in that structs (like https://github.com/Halium/android-headers/compare/halium-7.1#diff-ec4c7ddf4b1272307cfecf0412515d74L303) make apps using them (which talk directly to HWComposer) binary-incompatible with different HWComposer implementation on different Android versions. I suppose Android developers intention was to maintain binary-compatbility, as indicated by padding, but QTI_BSP ifdefs break that.

[ollieparanoid]

Thanks for the in-depth explanation!

It works the way you expect for libEGL/libGLESv2

How about we make kwin use libGLESv2 then? According to this announcement, it should work.

Do you think that would work? Maybe Nemo/LuneOS have similar options to choose a different backend, which would make it possible to avoid the device specific code (that wraps hwcomposer) altogether.

[bhush9]

How about we make kwin use libGLESv2 then? According to this announcement, it should work.

We can not directly use libGLESv2 without initializing the hwcomposer module in most of the devices.

[NotKit]

How about we make kwin use libGLESv2 then? According to this announcement, it should work.

What @bhush9 just said. libhybris provides Wayland windowing platform integration for apps, but Wayland compositor should use hwcomposer module to output on screen. I suppose proper long term solution would be to provide a wrapper for hwcomposer module structs in libhybris, so Android headers are not used directly, but I think it's better to get it running at all first.

[ollieparanoid]

Looks like you're past the "get it running at all" point now, quoting from your excellent summary of work in #1002:

It can start Xorg server with mentioned driver and utilize 3D acceleration via libhybris (tested with hildon-desktop).

So... you proposed "a wrapper for hwcomposer module structs in libhybris". That seems to be the right way forward to me with integrating this. But I have some questions, which I think are worth discussing first:

Packages (with the wrapper)

Let's say you had already written that wrapper, which packages would we need to have for the libhybris support then?

I'm thinking of ideally only having one libhybris package, that was built against all sorts of android headers, and uses the right one on the fly with the wrapper. We could store the right headers to use in a config file (...generated from the deviceinfo?). Does that make sense?

Build Android stuff from source?

Would it be possible to package all the components from Android as postmarketOS/Alpine packages, instead of downloading them as pre-built blob that comes out of Android's build system?

You mentioned, that they are low level components without JVM etc., so just in theory, if we wrote regular Makefiles for them, it should be possible? We've been trying to do some of that for android-rild, but it isn't very far and currently I'm not working on it. It would be nice, if we had a working android-rild packaged as side-effect of the libhybris porting effort if that makes sense.

LXC vs. side by side

If we build everything by ourselves, does the LXC approach make more sense (as done by Halium), or the one with running everything side by side with Linux (as done by SFOS)?

My opinion is, that if we can have single lightweight packages, I would rather not run Android's init system etc. in the LXC container, but start android-rild (and whatever else is needed) with simple OpenRC scripts and isolate/harden them on a per-application basis.

But if it turns out that this isn't feasible, I think LXC is the way to go.

---

So yeah... you have probably noted that Andorid internals/how libhybris works isn't something I understand well yet, but that's why I'm asking so many questions. 👍

[NotKit]

Packages (with the wrapper)
The idea of wrapper I proposed was to avoid packages utilizing HWComposer API for screen out having to be compiled once for each headers version (kwin, qt5-qpa-hwcomposer for LuneOS/Nemo, mine proof-of-concept xf86-video-hwcomposer and so on). So that still leaves different pairs of libhybris/headers packages. There was an attempt to allow building only non-depedant on headers libhybris core parts from one of its devs, but it never got merged: https://github.com/libhybris/libhybris/pull/302

Build Android stuff from source?
It should be technically possible, but I don't think it's feasible. At least before Oreo, Android got compiled per device as a rule, and there is no Android versions that fits all the devices. There are a lot of device-specific hacks and changes here and there, init scripts and so on. That's basically what makes CyanogenMod/LineageOS porting complicated (or building working system from Google-published AOSP sources for specific device). mer-hybris/Halium partly reuse the effort done for LineageOS porting by using it as base.

Halium was born as a project to provide common Android base for multiple distributions. In case of using Halium, there is going to be single per-device image (system.img) which is mounted inside LXC container. I thought of downloading it during post-install phase instead of packaging, as it's going to be pretty huge for newer devices (300+ mb).

LXC vs. side by side
To me, Halium LXC approach looks cleaner. But Mer/Sailfish probably have the most adaptations so far, so I supposed both approaches could be used depending on what is available for the device.

[ollieparanoid]

So that still leaves different pairs of libhybris/headers packages. There was an attempt to allow building only non-depedant on headers libhybris core parts from one of its devs, but it never got merged.

If that worked, that would be nice! But it looks like the author only tested if it compiled, so I doubt that it works out of the box.

In general, when shipping -4.4, -7.1 etc. packages for something, I would prefer if that was generated from one aport/APKBUILD, so it would be easier to update and maintain (I see that you've split libhybris in other ways, which is also good - we can probably combine that with an additional per-Android version split).

In fact, having all libhybris versions generated from one aport makes the maintenance overhead for the different headers so small that it almost doesn't matter.

All in all I think it is obvious that we need the hwcomposer wrapper to sanely integrate it with the other aports. When you wrote it above, it sounded more like an idea, but do you plan to work on it any time soon? If you (or someone else) would like to, we could also put up a repository in the postmarketOS GitHub project for this (so it gets more visibility and collaboration might become easier).

Thanks a lot for the explanations!

[ollieparanoid]

I have a new idea, but I am not sure if this works.

Let's say we added a hwcomposer backend to weston, and compiled that backend multiple times for each android headers version. Weston backends are nicely split into own *.so files, and even packaged separately in Alpine.

Would it be possible to run kwin_wayland/the luna compositor/X11 on top of weston and use the hardware 3D acceleration that way?

[NotKit]

They have support for running as nested compositors, but this is going to add extra overhead (running two compositors instead of just one, forwarding surfaces and input) and source of bugs. I don't think it's worth it.

[ollieparanoid]

Correct me if I'm wrong (I am really uncertain of this theory with my limited Wayland knowledge), but as I understand the actual rendering can be passed through super efficiently, citing from this commit:

Using this extension it can attach the client buffers directly to the subsurface without having to blit the contents into an intermediate buffer. The compositing can then be done in the parent compositor.

...so what's left is passing inputs such as touch screen presses/movements and the setup of the interfaces, which does not sound expensive compared to the actual rendering, right?

I've considered this theory, because we need to package kwin_wayland (and the luneOS compositor) on top of weston anyway, to get any software rendering on framebuffer with QtWayland (yes, bad performance, maybe improved software rendering helps). So we would have this anyway, and if we also did this with libhybris, we would have the cleanest libhybris packaging I can think of (only libhybris-* and weston-backend-hwcomposer-* for all the android versions, right?), which fits better in the projects vision of not-having android-version specific stuff (because at least we don't have as many packages then). If this is all accurate, the packaging benefits would make the efforts worth in my opinion.

Thanks for thinking these ideas through with me @NotKit , I'm interested if that changes your mind or not. And I'm also interested in what everyone else thinks about this.

[ollieparanoid]

@NotKit answered in the chat:

as for proxying via weston, the thing I don't like is that this trades runtime improvements for packaging improvements
it should not be much overhead in theory, but there were attempts of running nested compositors in Sailfish (which uses lipstick compositor => qt5-qpa-hwcomposer-plugin, similar to Nemo and Lune) and it didn't work quite right (input issues, screen corruption, missing rotation)
the biggest package that needs it (in terms of compilation time) is kwin, but HWComposer backend is still a separate lib there and it should be possible to compile different versions of it without rebuilding kwin
qt5-qpa-hwcomposer-plugin is rather small, since it just provides backend for Qt

[ollieparanoid]

Libhybris is merged (#1402) as well as the code changes to make proprietary firmware and userspace (which could be drivers working with libhybris) optional (#756). Closing.

[kskarthik]

Why not just include bionic in pmos instead of porting libhybris ? Please educate me.

[mirh]

Because just use Android at that point?
"Desktop vanilla" gnu/linux uses libc instead.

[kskarthik]

Alpine uses musl &/why don't ship bionic along with musl?