Python-for-android: Need libpython3.7m.so.1.0 in android phone

[homdx]

Hello.
I build with this instruction:

https://groups.google.com/forum/#!topic/kivy-users/Lf4zlYmLVPo

Apk compiled and builded. After setup and launch in my Android phone, I have a error:
```
SDL Error

An error occurred while trying to start the applicaion. Please try again and/or reinstall.
Error: dl open failed: could not load library: "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not foud

log from logcat:

V/pythonutil(11885): Checking pattern libcrystax.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libcrystax.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libcrystax.so against libmain.so
V/pythonutil(11885): Checking pattern libcrystax.so against libSDL2.so
V/pythonutil(11885): Checking pattern libcrystax.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libcrystax.so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libmain.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libSDL2.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libsqlite3.so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libssl..so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libssl..so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libssl..so against libmain.so
V/pythonutil(11885): Checking pattern libssl..so against libSDL2.so
V/pythonutil(11885): Checking pattern libssl..so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libssl..so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libcrypto..so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libcrypto..so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libcrypto..so against libmain.so
V/pythonutil(11885): Checking pattern libcrypto..so against libSDL2.so
V/pythonutil(11885): Checking pattern libcrypto..so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libcrypto..so against libpython3.7m.so
V/pythonutil(11885): Loading library: SDL2
V/pythonutil(11885): Loading library: SDL2_image
V/pythonutil(11885): Loading library: SDL2_mixer
V/pythonutil(11885): Loading library: SDL2_ttf
V/pythonutil(11885): Loading library: python2.7
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython2.7.so"
V/pythonutil(11885): Loading library: python3.5m
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython3.5m.so"
V/pythonutil(11885): Loading library: python3.6m
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython3.6m.so"
V/pythonutil(11885): Loading library: python3.7m
V/pythonutil(11885): Loading library: main
E/art (11885): dlopen("/data/app/com.kivymd.test-1/lib/arm/libmain.so", RTLD_LAZY) failed: dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found
V/pythonutil(11885): Library loading error: dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found
V/pythonutil(11885): An UnsatisfiedLinkError occurred loading main
W/System.err(11885): dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found```

[etc0de]

It seems like something with the linker options might have gone wrong, so it might be helpful to see the build. Could you enable p4a's --debug option, rerun the build, and then attach the output as a text file? (It's going to be very long)

[homdx]

build.log

[etc0de]

@inclement could you take a look at this? This is something related to the new python3 recipe, and it seems main.so (the start.c bootstrap entry point of the sdl2 bootstrap) is built wrong.

This is the main.so linker line from the log:

[DEBUG]:    [armeabi-v7a] SharedLibrary  : libmain.so
[DEBUG]:    /home/user/hostcwd/manbuild/android-ndk-r17c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -Wl,-soname,libmain.so -shared --sysroot=/home/user/hostcwd/manbuild/android-ndk-r17c/platforms/android-21/arch-arm /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/objs/main/__/__/SDL/src/main/android/SDL_android_main.o /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/objs/main/start.o -lgcc -Wl,--exclude-libs,libgcc.a -latomic -Wl,--exclude-libs,libatomic.a /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/libSDL2.so  -gcc-toolchain /home/user/hostcwd/manbuild/android-ndk-r17c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -no-canonical-prefixes -target armv7-none-linux-androideabi21 -Wl,--fix-cortex-a8 -L/home/user/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/android-build -Wl,--build-id -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--warn-shared-textrel -Wl,--fatal-warnings   -L/home/user/hostcwd/manbuild/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib -lGLESv1_CM -lGLESv2 -llog -lpython3.7m -lc -lm -o /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/libmain.so

I suspect the -lpython3.7m is potentially the issue. I wonder, is clang linking the full name libpython3.7m.so.1.0 when it should just link libpython3.7m.so (which was already loaded as per the runtime log in the original post, Loading library: python3.7m)? Might we need either an option for clang to not do that, or a symlink of that full name back to the short named actual binary?

Maybe this is an issue with the specific newer/older clang from this particular NDK version?

[inclement]

I'm not sure exactly what is wrong here, but both libpython3.7m.so and libpython3.7m.so.1.0 should be present in the APK. I think it is expected that libpython3.7m.so relies on symbols loaded from libpython3.7m.so.1.0. I wondered if this would cause issues with linking when I first set this up, but forgot about it when it didn't appear to do so.

I wonder, does this still happen if libpython3.7m.so.1.0 is loaded explicitly when the app is started, as we do for the other library files? I think this would require slightly more modification to PythonUtil.java than one might hope, because the Java method called to load the libraries automatically finds them and only supports a .so suffix (although I could be wrong about that).

[homdx]

@inclement @JonasT You can reproduce this? (This error launch in phone)?

--dist_name=py3-sink
--private .
--package=com.kivymd.test
--name KivyMD
--bootstrap=sdl2
--requirements=python3,kivy
--arch=armeabi-v7a
--icon /home/user/hostcwd/app/kivymd/images/kivymd_logo.png
--sdk_dir /home/user/hostcwd/manbuild/sdk
--ndk_dir /home/user/hostcwd/manbuild/android-ndk-r17c
--android_api 28
--ndk_version 17c
--ndk-api 21
--version 0.0.1

Also this error aviable, then I compile Hello World. This apk in phone start with error (see bottom message) :-(

[homdx]

py3-sink.zip

You can uncompress for sea this result apk

[etc0de]

I just checked, there is only lib/armeabi-v7a/libpython3.7m.so - no trace of a libpython3.7m.so.1.0

[inclement]

Ah, quite interesting. I wonder at what point in the build process it failed to appear or be copied around.

@homdx it looks like you are using p4a directly? What files are present in `~/.local/share/python-for-android/dists/$YOURDISTNAME/libs/armeabi-v7a?

[homdx]

I can reproduce for you Docker file, if you need?)

[homdx]

yes. I run with:
$p4a apk
@JonasT @inclement
compiled files:

:~/.local/share/python-for-android/dists/py3-sink$ ls -la libs/armeabi-v7a/
total 13820
drwxr-xr-x. 2 user user     164 Dec  2 07:09 .
drwxr-xr-x. 3 user user      25 Dec  2 07:07 ..
-rwxr-xr-x. 1 user user   18112 Dec  2 07:09 libmain.so
-rwxr-xr-x. 1 user user 2776024 Dec  2 07:09 libpython3.7m.so
-rwxr-xr-x. 1 user user 9357004 Dec  2 07:08 libpython3.7m.so.1.0
-rwxr-xr-x. 1 user user  415396 Dec  2 07:09 libSDL2_image.so
-rwxr-xr-x. 1 user user  282740 Dec  2 07:09 libSDL2_mixer.so
-rwxr-xr-x. 1 user user  878352 Dec  2 07:09 libSDL2.so
-rwxr-xr-x. 1 user user  407204 Dec  2 07:09 libSDL2_ttf.so

[etc0de]

@homdx I just attempted to configure your exact build combination and see if compiling the kivy demo app will include libpython3.7m.so.1.0 for me or not, but I ran into another error #1502 - oops!

Basically all I'm saying is I'll tell you if I can reproduce this as soon as I manage to actually get through the build :laughing: hang in there

[etc0de]

@inclement I can confirm that libpython3.7m.so.1.0 isn't added even for a trivial test app:

Steps to reproduce manually:

1. Install NDK r17c (NDK API 21), SDK 28
2. Install p4a master, python3, cython and such
3. Fetch https://github.com/kivy/python-for-android/tree/master/testapps/testapp_keyboard
4. Build test app with: p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private /path/to/app and all the necessary options to use your NDK r17c with API 21, and the Android SDK with API 28 with their given paths and such

Steps to reproduce in docker environment: (with my little experimental build environment tool)

1. Install docker, launch docker daemon
2. sudo pip3 install https://github.com/JonasT/p4a-build-spaces/archive/master.zip (installs p4a-build-spaces, system-wide to get the binary installed to $PATH)
3. sudo p4aspaces shell p4a-py3-api28ndk21 --p4a master (launches the build env - sudo only needed if your regular user isn't in the docker group)
4. In the build shell: cd ~/testapp && p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private . (no more options required, SDK/NDK are preconfigured via env)

This is the result when examining the .apk (note the missing libpython3.7m.so):

# unzip ../unnamed_dist_1-1-debug.apk 
Archive:  ../unnamed_dist_1-1-debug.apk
  inflating: AndroidManifest.xml     
  inflating: META-INF/CERT.RSA       
  inflating: META-INF/CERT.SF        
  inflating: META-INF/MANIFEST.MF    
 extracting: assets/private.mp3      
  inflating: classes.dex             
  inflating: lib/armeabi-v7a/libSDL2.so  
  inflating: lib/armeabi-v7a/libSDL2_image.so  
  inflating: lib/armeabi-v7a/libSDL2_mixer.so  
  inflating: lib/armeabi-v7a/libSDL2_ttf.so  
  inflating: lib/armeabi-v7a/libmain.so  
  inflating: lib/armeabi-v7a/libpython3.7m.so  
 extracting: res/drawable-hdpi-v4/ic_launcher.png  
 extracting: res/drawable-mdpi-v4/ic_launcher.png  
 extracting: res/drawable-xhdpi-v4/ic_launcher.png  
 extracting: res/drawable-xxhdpi-v4/ic_launcher.png  
 extracting: res/drawable/icon.png   
 extracting: res/drawable/presplash.jpg  
  inflating: res/layout/chooser_item.xml  
  inflating: res/layout/main.xml     
  inflating: res/layout/project_chooser.xml  
  inflating: res/layout/project_empty.xml  
 extracting: resources.arsc

However, the .so is still present in the original dist folder:

# ls /root/.local/share/python-for-android/dists/unnamed_dist_1/libs/armeabi-v7a/
libSDL2.so  libSDL2_image.so  libSDL2_mixer.so  libSDL2_ttf.so  libmain.so  libpython3.7m.so  libpython3.7m.so.1.0

[inclement]

Wow, that's quite interesting. Am I right in thinking that you've successfully used the python3 recipe, so it isn't a generic problem that happens to everyone except me?
I wonder if there's some gradle setting changed or similar, affecting what libraries get bundled.
I'd hope that it should be simple enough to make gradle include the file, unless it has some specific reason that it doesn't want to. What version of the build-tools is this using?

[etc0de]

@inclement the exact environment p4a-py3-api28ndk21's details are specified here:

https://github.com/JonasT/p4a-build-spaces/blob/master/environments/p4a-py3-api28ndk21/Dockerfile

Edit: I don't know if it's a generic problem, because I haven't tried any other environment with the python3 recipe. I use python3crystax with the TLS hack rebuild usually, which has no such issues but I guess that's also packaged differently

Please also note that sudo p4aspaces shell p4a-py3-api28ndk21 --p4a master gives you, after some build waiting time, a bash with everything isolated & installed in the affected versions in docker (needs docker installed of course). Just in case that would help with debugging. I can also assist with that on discord if that helps, both debugging and launching the environment

[homdx]

@JonasT Thank you. You release very interesting tool p4a-build-spaces:

Step 27/27 : CMD bash -c "`python3 /cmdline.py`"
 ---> Running in 1db4bc5c74d7
 ---> ae0bb8e5ac24
Removing intermediate container 1db4bc5c74d7
Successfully built ae0bb8e5ac24

To build a kivy demo app, use this command:
cd ~/testapp && p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private .
root@c528c50f1cd9:~#

Now wait fix by @inclement for python p4a apk build

[maho]

while trying to build with Dockerfile.py3 - I have identical error msg.

cmd within docker is

~/venv/bin/p4a apk --sdk-dir=/opt/android/android-sdk/ --android-api=27 --ndk-dir=/opt/android/android-ndk-r16b/ --ndk-version=16b --requirements=python3,kivy --package=maho.test --name=pong --version=0.0.1 --private=$PWD --debug --copy-libs

[maho]

short investigation: gradle is guilty for such situation. Why? it's total mysterious tool for me, so I have no idea. However - so.1.0 exists in libs/ dir, but is absent in .apk

[opacam]

Ok, I solved the linkage problem with libpython3.7.so.1.0, and @inclement was in the right direction:

...the Java method called to load the libraries automatically finds them and only supports a .so suffix

So...there is no way that the java loader can load a versioned library...therefore... is no problem with gradle, is our build method what is causing the problems, but this can be easily solved by forcing that the python library gets built without the version, so this way we will have the proper suffix, and we will avoid all this linkage problems.

The fix is already applied in pr #1537, but here I leave a link to a patch that will solve the problem for the current master branch: https://gist.github.com/opacam/10cef1ca8182a39b9e5ebffe0adcdc90#file-fix-python3-linkage-patch

[homdx]

@opacam I comfimed. Now app are start in mobile phone without this error)

I/python  (20877): [INFO   ] [Kivy        ] v1.10.1
I/python  (20877): [INFO   ] [Python      ] v3.7.1 (default, Jan  7 2019, 21:35:27) 
I/python  (20877): [Clang 5.0.300080 ]

But I compile only with buildozer 0.37: api = 27, sdk=24, nkd 16b,
How use and build with p4a ? Wait merge to master branch python-for-android ?
Or you can add instruction for me?

[homdx]

@opacam Thank you! I pushed you patch for testing in branch fix/python37
https://github.com/homdx/p4a-build-spaces
@JonasT, @inclement You can probe build )

[opacam]

How use and build with p4a ?

My method consist in set the variable p4a.source_dir of my app's buildozer spec file to point to my pythonforandroid development folder, so I can build with any branch of my local fork of p4a...but maybe someone has a better method...

[homdx]

@opacam I think, need feature for p4a with path like p4a.source_dir in buildozer

[opacam]

This option is commented in the default buildozers spec file...I suppose that you modified this file from a previous version of buildozers...so...if you don't find this option commented in your buildozers file just add it and put the corresponding path (look at almost the end of the section app, before buildozer)

[tito]

Guys, about this, previous Python 2.7 compilation was forcing INSTSONAME to be just libpython2.7.so, while now, none of the Python 2 and 3 recipes set it.
In consequence, the versionned version of the library will be used for linkage for all the library that do -lpython.

The right fix is either set INSTSONAME=libpythonXX.so as before, either patch Python's configure to remove the versionning (like i do in https://github.com/kivy/python-for-android/commit/15a3770b403f82ff04c7681449758133ec2cfec8). Then when compiling, it will use just libpythonXX.so, and everything will be fine.

[etc0de]

This might not be actually fixed, since the commit was backed out into the separate #1568 mostly because I requested it (because I asked to not bloat #1537 further). So I think this issue still persists, and will be fixed by #1568

[KeyWeeUsr]

Merged #1568. Please test with the current master if the error is still present.