Pmbootstrap: Anbox support

Created on 4 Mar 2018  路  18Comments  路  Source: postmarketOS/pmbootstrap

Back in August, @MartijnBraam started packaging Anbox but got stuck somewhere in the process. I just rebased the branch to the latest master and updated some dependencies.

However one of it's dependencies, dbus-cpp, fails to compile (we package it ourselves). We're guessing it's a musl problem, which might make this quite hard to resolve.

[  4%] Building CXX object src/core/dbus/CMakeFiles/dbus-cpp.dir/fixture.cpp.o
/home/pmos/build/src/build/src/core/dbus/fixture.cpp:26:29: fatal error: core/posix/exec.h: No such file or directory
 #include <core/posix/exec.h>
                             ^
compilation terminated.
make[2]: *** [src/core/dbus/CMakeFiles/dbus-cpp.dir/build.make:63: src/core/dbus/CMakeFiles/dbus-cpp.dir/fixture.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1451: src/core/dbus/CMakeFiles/dbus-cpp.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
help wanted packaging

Most helpful comment

Slowly making progress, here's the first screenshot!

anbox_wip

That's the part running as user and it's saying in the terminal that it can't connect to the session daemon and then gives up after a few seconds, but still it's satisfying to have at least some visual :grin:

Status:

  • Container manager starts, mounts the android image and runs in background
  • Session manager doesn't run yet: it requests a new container from container manager, and the container can't be set up because of an error coming from lxc (it would fail earlier because a fifo can not be set up, but I have a workaround for that now).

lxc_cgfs - cgroups/cgfs.c:cgfs_init:2363 - cgroupfs failed to detect cgroup metadata

I've added a temporary README.anbox file to the branch with instructions on how to start everything, how to debug, where the log file is and what is left to do: https://github.com/postmarketOS/pmbootstrap/blob/feature/anbox/README.anbox

All 18 comments

A quick apt-file search core/posix/exec.h querry on debian suggests that the dependency process-cpp is missing

Good call, thanks for that!

Some new errors though, this time in process-cpp :unamused:

[ 11%] Building CXX object src/CMakeFiles/process-cpp.dir/core/posix/child_process.cpp.o
In file included from /usr/include/fortify/stdio.h:20:0,
                 from /usr/include/c++/6.4.0/cstdio:42,
                 from /usr/include/c++/6.4.0/ext/string_conversions.h:43,
                 from /usr/include/c++/6.4.0/bits/basic_string.h:5417,
                 from /usr/include/c++/6.4.0/string:52,
                 from /usr/include/c++/6.4.0/stdexcept:39,
                 from /usr/include/c++/6.4.0/array:39,
                 from /usr/include/c++/6.4.0/tuple:39,
                 from /usr/include/c++/6.4.0/functional:55,
                 from /usr/include/core/connection.h:21,
                 from /usr/include/core/signal.h:21,
                 from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/signal.h:24,
                 from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/signalable.h:22,
                 from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/process_group.h:22,
                 from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/process.h:22,
                 from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/child_process.h:22,
                 from /home/pmos/build/src/process-cpp-3.0.1/src/core/posix/child_process.cpp:19:
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:36:5: error: expected identifier before '(' token
     stdin = 1 << 0,
     ^
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:36:5: error: expected '}' before '(' token
In file included from /home/pmos/build/src/process-cpp-3.0.1/include/core/posix/child_process.h:23:0,
                 from /home/pmos/build/src/process-cpp-3.0.1/src/core/posix/child_process.cpp:19:
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:36:18: error: cannot convert 'int' to 'core::posix::StandardStream' in initialization
     stdin = 1 << 0,
                  ^
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:37:19: error: cannot convert 'int' to 'core::posix::StandardStream' in initialization
     stdout = 1 << 1,
                   ^
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:38:19: error: cannot convert 'int' to 'core::posix::StandardStream' in initialization
     stderr = 1 << 2
                   ^
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:39:2: error: extra ';' [-Werror=pedantic]
 };
  ^
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:41:23: error: 'StandardStream' does not name a type
 CORE_POSIX_DLL_PUBLIC StandardStream operator|(StandardStream l, StandardStream r);
                       ^~~~~~~~~~~~~~
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:42:23: error: 'StandardStream' does not name a type
 CORE_POSIX_DLL_PUBLIC StandardStream operator&(StandardStream l, StandardStream r);
                       ^~~~~~~~~~~~~~
/home/pmos/build/src/process-cpp-3.0.1/include/core/posix/standard_stream.h:44:1: error: expected declaration before '}' token
 }
 ^
cc1plus: all warnings being treated as errors
make[2]: *** [src/CMakeFiles/process-cpp.dir/build.make:87: src/CMakeFiles/process-cpp.dir/core/posix/child_process.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1036: src/CMakeFiles/process-cpp.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

could you share the APKBUILD?

EDIT: never mind, it's in the feature/anbox branch. If someone wants to join in with fixing the build errors:

git checkout -b feature/anbox origin/feature/anbox
pmbootstrap build process-cpp
# build fails but downloads sources
cd ~/code
tar -xvf ~/.local/var/pmbootstrap/cache_distfiles/process-cpp_3.0.1.orig.tar.gz
cd process-cpp-3.0.1
git init .
git add -A
git commit -m "initial"

Then repeat until everything is fixed:
```

fix something

git commit -m "fixed xyz"
pmbootstrap build process-cpp --src=.
````

I don't really know how to fix it:

.../include/core/posix/standard_stream.h:36:5: error: expected identifier before '(' token
     stdin = 1 << 0,
     ^

There is no ( character in that line, gcc! My guess is that this comes from a macro?

I've also looked at Debian's patches for this, but nothing that helps with this issue:

debian/patches/
debian/patches/0001-Don-t-run-tests.patch
debian/patches/0002-Reproducible-documentation.patch

@PureTryOut: Do you want to try asking about that on stack overflow or C++ IRC/matrix channels?

It looks like if I comment out these lines in /usr/include/stdio.h, some of the errors go away:

#define stdin  (stdin)
#define stdout (stdout)
#define stderr (stderr)

That would be the macro inserting a ( as @ollieparanoid surmised. Apparently the standard does say these symbols should be macros, but my GNU libc stdio.h defines them without parens. The parens muck up the enum definition, I think.

Maybe the best solution is inserting your own stdio.h into process-cpp's internal include path (don't install it to the system), which does something like this (untested):

#pragma once

#include_next <stdio.h>
#undef stdin
#undef stdout
#undef stderr
#define stdin  stdin
#define stdout stdout
#define stderr stderr

When the parens are gone, I don't think there will be adverse effects, but that enum at standard_stream.h:36 should compile.

Thanks for the hint @scintill, that helped so much! I ended up adding 3 #undef line just before the namespace that failed to compile in standard_stream.h and that did the trick. With linking against execinfo to fix undefined reference errors for backtrace stuff (which is present in glibc, but not in musl) and fixing one uint -> unsigned int, the process-cpp package compiles now.

dbus-cpp was an easy fix as well, I only needed to change the subpackages.
Now it starts compiling anbox and fails because lxc is missing or something.
I've rebased the branch and applied the fixes.

@PureTryOut, do you want to give it another try?

Awesome job! Up to the next error, we're close now!

[100%] Linking CXX executable anbox
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot open output file anbox: Is a directory
collect2: error: ld returned 1 exit status

Note that I disabled unit tests entirely, as it requires a framework called gmock which I failed to get to install as of yet. I might look into it again later, but I feel it isn't as important.

@ollieparanoid suggested trying to build outside of the source directory. That did seem to resolve above linking error, but a new one occurs.

[100%] Linking CXX executable anbox
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: ../external/process-cpp-minimal/src/libprocess-cpp.a(backtrace.cpp.o): undefined reference to symbol 'backtrace_symbols'
//usr/lib/libexecinfo.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/anbox.dir/build.make:119: src/anbox] Error 1
make[1]: *** [CMakeFiles/Makefile2:2134: src/CMakeFiles/anbox.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Also, using -j1 (which I did for above) gives me vastly different results than not specifying -j at all.
When doing the latter I get the following:

[ 31%] Linking CXX static library librenderControl_dec.a
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLEScontext.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/GLEScontext.cpp:17:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:18:56: fatal error: OpenGLESDispatch/gles_extensions_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles_extensions_functions.h"
                                                        ^
compilation terminated.
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:111: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/GLEScontext.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/GLDispatch.cpp:17:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:18:56: fatal error: OpenGLESDispatch/gles_extensions_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles_extensions_functions.h"
                                                        ^
compilation terminated.
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLEScontext.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLESvalidate.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/GLESvalidate.cpp:17:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:18:56: fatal error: OpenGLESDispatch/gles_extensions_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles_extensions_functions.h"
                                                        ^
compilation terminated.
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:63: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/GLDispatch.cpp.o] Error 1
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:135: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/GLESvalidate.cpp.o] Error 1
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLEScontext.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/TextureUtils.h:21,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/TextureUtils.cpp:16:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:18:56: fatal error: OpenGLESDispatch/gles_extensions_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles_extensions_functions.h"
                                                        ^
compilation terminated.
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:231: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/TextureUtils.cpp.o] Error 1
[ 31%] Built target renderControl_dec
Generating /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/build/external/android-emugl/host/include/OpenGLESDispatch/gles1_only_functions.h
[ 31%] Built target xdg_test
Generating /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/build/external/android-emugl/host/include/OpenGLESDispatch/gles1_extensions_functions.h
Generating /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/build/external/android-emugl/host/include/OpenGLESDispatch/gles2_only_functions.h
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLEScontext.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/objectNameManager.cpp:18:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:21:51: fatal error: OpenGLESDispatch/gles2_only_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles2_only_functions.h"
                                                   ^
compilation terminated.
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:303: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/objectNameManager.cpp.o] Error 1
Generating /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/build/external/android-emugl/host/include/OpenGLESDispatch/gles2_extensions_functions.h
In file included from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLDispatch.h:23:0,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/include/GLcommon/GLEScontext.h:20,
                 from /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/libs/Translator/GLcommon/FramebufferData.cpp:19:
/home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/external/android-emugl/host/include/OpenGLESDispatch/gles_functions.h:22:57: fatal error: OpenGLESDispatch/gles2_extensions_functions.h: No such file or directory
 #include "OpenGLESDispatch/gles2_extensions_functions.h"
                                                         ^
compilation terminated.
make[2]: *** [external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/build.make:327: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/FramebufferData.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1573: external/android-emugl/host/libs/Translator/GLcommon/CMakeFiles/GLcommon.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Generating /home/pmos/build/src/anbox-03f69c8685e8e05a8c296896431d0645da831092/build/external/android-emugl/host/include/OpenGLESDispatch/gles3_only_functions.h
[ 32%] Linking CXX static library libanbox-protobuf.a
[ 32%] Built target GLHeaders
[ 32%] Built target anbox-protobuf
make: *** [Makefile:130: all] Error 2

@PureTryOut: could you upload your latest version uploaded to the feature/anbox branch? I've just tried it, but I hit the cannot open output file anbox: Is a directory error which is already solved in your latest post.

@ollieparanoid done. Also rebased to latest master for good measure.

Thanks! You were pretty close already, with some more fixing it compiles now :smiley:
Back to you, @PureTryOut!

@PureTryOut: what was the status on this? In case we could get it working in time, it would be nice for the 1 year blog post as well.

That would be awesome indeed. I believe there are just a few things left to be done:

  • write an OpenRC service file, so far there are only systemd ones out there
  • write a .desktop file
  • create a basic Android image to use in Anbox

I don't have any experience with OpenRC init files really. I could learn how to do it, but it might be better if someone else tried.
We can take the .desktop file from Arch here.
For the Android image we could use the pre-built one by the Anbox project, but I'm not sure if we want that (we probably do as building our own image has some insane pc requirements).

EDIT: It seems I had a .desktop file locally, just uncommitted. I pushed it to the branch.

Right now it fails to build again, it seems some Boost libraries are updated. Some C++ experts out here?

In file included from /home/pmos/build/src/anbox-f68725cff813700e448fb3b2f3d57d88e2c7daae/src/anbox/cmds/session_manager.cpp:56:0:
/usr/include/core/dbus/asio/executor.h:29:7: error: using typedef-name 'boost::asio::io_service' after 'class'
 class io_service;
       ^~~~~~~~~~
In file included from /usr/include/boost/asio.hpp:69:0,
                 from /home/pmos/build/src/anbox-f68725cff813700e448fb3b2f3d57d88e2c7daae/src/anbox/runtime.h:21,
                 from /home/pmos/build/src/anbox-f68725cff813700e448fb3b2f3d57d88e2c7daae/src/anbox/audio/server.h:21,
                 from /home/pmos/build/src/anbox-f68725cff813700e448fb3b2f3d57d88e2c7daae/src/anbox/cmds/session_manager.cpp:26:
/usr/include/boost/asio/io_service.hpp:27:20: note: 'boost::asio::io_service' has a previous declaration here
 typedef io_context io_service;
                    ^~~~~~~~~~
make[2]: *** [src/CMakeFiles/anbox-core.dir/build.make:1363: src/CMakeFiles/anbox-core.dir/anbox/cmds/session_manager.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2088: src/CMakeFiles/anbox-core.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

As discussed in #postmarketOS, I think it would be good to make it work with the pre-built Android images first (possibly look into building our own ones later if it makes sense). That leaves the OpenRC service and .desktop files. Could you take a shot at them?

Build errors are fixed now, with patches from @tmn505 posted in https://github.com/anbox/anbox/issues/637 (thanks a lot for those!).

@PureTryOut: updated the branch. make sure to force rebuild dbus-cpp first:

$ pmbootstrap build --strict --force dbus-cpp
$ pmbootstrap build --strict anbox

Slowly making progress, here's the first screenshot!

anbox_wip

That's the part running as user and it's saying in the terminal that it can't connect to the session daemon and then gives up after a few seconds, but still it's satisfying to have at least some visual :grin:

Status:

  • Container manager starts, mounts the android image and runs in background
  • Session manager doesn't run yet: it requests a new container from container manager, and the container can't be set up because of an error coming from lxc (it would fail earlier because a fifo can not be set up, but I have a workaround for that now).

lxc_cgfs - cgroups/cgfs.c:cgfs_init:2363 - cgroupfs failed to detect cgroup metadata

I've added a temporary README.anbox file to the branch with instructions on how to start everything, how to debug, where the log file is and what is left to do: https://github.com/postmarketOS/pmbootstrap/blob/feature/anbox/README.anbox

I've added a service file, some post-install and post-deinstall hooks and made the loop kernel module autoload, so step 1 in README.anbox now happens automatically on boot.

However, I don't get to the point you have gotten @ollieparanoid (using the instructions in README.anbox). When I try to load anbox session-manager I just get the following:

[ 2018-04-30 14:33:27] [client.cpp:49@start] Failed to start container: Failed to start container: Failed to start container
[ 2018-04-30 14:33:27] [session_manager.cpp:162@operator()] Lost connection to container manager, terminating.

Nothing happens on the screen either. Also, the result isn't any different if I don't create the fifo file.

Thanks for the streamlining! Yeah, the session manager does not work yet, I get the same output and from anbox session-manager. And /var/lib/anbox/logs/container.log has the real error message:

Current fatal error in step 4:
lxc_cgfs - cgroups/cgfs.c:cgfs_init:2363 - cgroupfs failed to detect cgroup metadata

The loading screen appears, when you just ignore that and try to launch an Android app anyway (and of course it fails because session manager is not running):

$ export DISPLAY=:0
$ anbox launch --package=org.anbox.appmgr --component=org.anbox.appmgr.AppViewActivity

My understanding is, that the container manager uses the LXC library to set up the containers. And it passes a bunch of configuration variables to LXC (which is quite readable when looking at the anbox source actually), and one of them is lxc.cgroup.use. This variable is empty, and then in the LXC code lxc_cgroup_load_meta() fails because it is empty. I had something along the lines of that noted down, but I didn't have time yet to look into it further.

Also right now we are running anbox in privileged mode, which means the container manager and session manager need root rights as I understand, but we could also make it run without root rights which would be better of course. The official snap packaging uses the non-privileged mode as well from what I can tell. In the anbox code, it passes different configuration options to LXC, so this might even avoid the error above.

Finally, another idea for debugging I had was installing Ubuntu in a VM and running anbox there as snap (which is the supported installation method by upstream), then figuring out what the value for lxc.cgroup.use is in Ubuntu, so we can emulate it. The config options may also be set by a global config file or something, so I'm not entirely sure how it would work in Ubuntu.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

ollieparanoid picture ollieparanoid  路  5Comments

zenety picture zenety  路  5Comments

ghost picture ghost  路  4Comments

erhoof picture erhoof  路  4Comments

cmdr2 picture cmdr2  路  7Comments