Termux-packages: Package request: python3-apt

[pschmitt]

Package description
This package provides APT python bindings required for Ansible. Without this, the apt: module cannot be used.

Link to home page and sources

1. Home page: https://packages.debian.org/search?keywords=python-apt
2. Source code: https://salsa.debian.org/apt-team/python-apt

Additional information
Have you compiled or tried to compile the package on device?

Yes. Without any patches the build fails with:

[...]
In file included from python/acquire.cc:26:
python/apt_pkgmodule.h:14:10: fatal error: 'apt-pkg/hashes.h' file not found
#include <apt-pkg/hashes.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.
error: command 'aarch64-linux-android-clang' failed with exit status 1

NOTE: The headers are provided by libapt-pkg-dev (not available on Termux either)

[Grimler91]

I've added a new package libapt-pkg, please give building on device another shot.
Note that you probably need to use the 1.4 branch of python-apt, as we have apt 1.4.9.

[xeffyr]

libapt-pkg is no longer separate package and is part of apt (splitting it makes apt unusable).

[pschmitt]

Thanks a lot guys!

I just tried what you suggested @Grimler91 and it works just fine (python-apt v1.4.0-beta).

ansible -i inventory.yml termux_remote -m apt -a "name=jq state=present"
termux_remote | SUCCESS => {
    "cache_update_time": 0,
    "cache_updated": false,
    "changed": false
}

Now I just need to figure out how to package a python library for Termux ;)
Should not be too hard since this is pretty much all that's required:

git clone --single-branch --branch 1.4.0_beta3 https://salsa.debian.org/apt-team/python-apt
cd python-apt
python setup.py install

Is there any package that is currently packaged similarly?

[pschmitt]

libapt-pkg is no longer separate package and is part of apt (splitting it makes apt unusable).

libapt-pkg still shows up when searching via apt. Is that intended?

[pschmitt]

asciinema seems fairly similar - with the exception that we need access to Python.h at build time.
I tried fiddling with LD_LIBRARY_PATH and PYTHONHOME - but that didn't get me anywhere. Is there something obvious that I am missing here?

See https://github.com/pschmitt/termux-packages/blob/python3-apt/packages/python3-apt/build.sh

[xeffyr]

libapt-pkg still shows up when searching via apt Is that intended?

I didn't removed previous versions of apt package, so assume that it is "intended".
Will remove it now.

[xeffyr]

I tried fiddling with LD_LIBRARY_PATH and PYTHONHOME - but that didn't get me anywhere

Building python modules (with exception for ones without native extensions) is unsupported by termux-packages.

Just python3.8 setup.py install --prefix=$TERMUX_PREFIX --force will not work in your build.sh script because it executes host python3.8 and not Termux's one which is cross-compiled and cannot be executed for host.

[Grimler91]

@xeffyr thanks for cleaning up my mess!

Running the host's python3.8 should be fine, that's what we do for asciinema and electrum. I'm more thinking that the Extension() call in setup.py needs to be patched to have the argument include_dirs = ['/data/data/com.termux/files/usr/include'], as done for example here: https://stackoverflow.com/a/10867041

[pschmitt]

@xeffyr thanks for cleaning up my mess!

Running the host's python3.8 should be fine, that's what we do for asciinema and electrum. I'm more thinking that the Extension() call in setup.py needs to be patched to have the argument include_dirs = ['/data/data/com.termux/files/usr/include'], as done for example here: https://stackoverflow.com/a/10867041

Thanks! That was exactly what I was looking for.
I actually managed to build a package this time, but it's a bit hackish (esp. the hardcoded /home/builder/.termux-build/apt/build/include path)

-> See https://github.com/pschmitt/termux-packages/blob/710ba64eef7883a9366e63bb31639f2addff85a0/packages/python3-apt/build.sh#L20

[xeffyr]

Isn't needed apt's include files available in $TERMUX_PREFIX/include ?
Latest apt package contains headers for apt-pkg, so you may try to use includes from $TERMUX_PREFIX instead of hardcoding builder path.

[pschmitt]

My bad. I didn't pull from origin/master beforehand. I removed the hardcoded path and updated my branch. Now I am able to build a package without any further fiddling. The only thing that I need to address now is that I end up with apt_inst.cpython-38-x86_64-linux-gnu.so and apt_pkg.cpython-38-x86_64-linux-gnu.so in the deb instead of arm libs - which obviously doesn't work. Is that what you were referring to @xeffyr with your comment about cross compilation?

[pschmitt]

If anyone is trying to get python-apt to work on termux the easiest (and only) way atm is:

pip install -e "git+https://salsa.debian.org/apt-team/[email protected]_beta3#egg=python-apt"

[xeffyr]

Is that what you were referring to @xeffyr with your comment about cross compilation?

Yes. Python obtains all information about correct compiler and flags from sysconfig module. It seems to ignore $CC.

[pschmitt]

Hm. Judging by the output of the build command it actually does use aarch64-linux-android-clang:

./build-package.sh -i python3-apt
[...]
running build_ext                                    building 'apt_pkg' extension                         creating build/temp.linux-x86_64-3.8                 creating build/temp.linux-x86_64-3.8/python          aarch64-linux-android-clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fstack-protector-strong -Oz -I/data/data/com.termux/files/usr/include -fPIC -I/data/data/com.termux/files/usr/include -I/data/data/com.termux/files/usr/include/python3.8 -I/usr/include/python3.8 -c python/acquire.cc -o build/temp.linux-x86_64-3.8/python/a
cquire.o -std=c++11 -Wno-write-strings -DAPT_8_CLEANER_HEADERS -DAPT_9_CLEANER_HEADERS -DAPT_10_CLEANER_HEADERS
[...]

[pschmitt]

Update: the resulting .so files are ARM binaries:

file *.so                                                                                                                                              
apt_inst.cpython-38-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
apt_pkg.cpython-38-x86_64-linux-gnu.so:  ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

Which sounds good, but even after creating symlinks manually, import apt fails:

cd /data/data/com.termux/files/usr/lib/python3.8/site-packages
ln -s apt_pkg.cpython-38-x86_64-linux-gnu.so apt_pkg.so
ln -s apt_inst.cpython-38-x86_64-linux-gnu.so apt_inst.so
python -c "import apt"                                                       
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ImportError: dlopen failed: cannot locate symbol "PyExc_ValueError" referenced by "/data/data/com.termux/files/usr/lib/python3.8/site-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so"...