Pyo3: Cross compile linking issue: cannot locate symbol "_Py_NoneStruct"

[Progdrasil]

🐛 Bug Description

I need to cross compile a pyo3 library for android using kivy/buildozer. I've set the LD_LIBRARY_PATH, PYO3_CROSS_INCLUDE_DIR and PYO3_CROSS_LIB_DIR environment variables and am using cargo ndk for the build process and copying the .so to the site_packages directory.

Everything compiles correctly, unfortunately at runtime I get the message cannot locate symbol "_Py_NoneStruct".

🌍 Environment

• Your operating system and version: Ubuntu 20.04 (docker)
• Your python version: 3.8
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: compiled from source
• Your Rust version (rustc --version): 1.45.0
• Your PyO3 version: 0.11.1
• Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: yes

💥 Reproducing

I have a repo for the minimal working example. The necessary commands are shown in the README.md

[davidhewitt]

Thanks for posting. This looks like a packaging issue, possibly in kivy/python-for-android. Here's an example of a very similar looking upstream bug from 2016 which was closed. https://github.com/kivy/python-for-android/issues/887

As they will have much better knowledge than I for diagnosing the buildozer config, I suggest reporting upstream there before I attempt any debugging here.

[Progdrasil]

I'll check that out, however I did have a similar issue when compiling the same project in termux I'm not sure if this constitutes a different issue, but it is the same target.

ImportError: dlopen failed: cannot locate symbol "PyExc_BaseException" referenced by "/data/data/com.termux/files/home/pyo3-buildozer/target/debug/my_lib.so"

The target is aarch64-linux-android

[Progdrasil]

After reading through the comments it seems what has fixed it was adding the LDFLAG+="-lpythonX.Ym" flag. I did get a linking issue when removing the extension-module where the linking flags had -lpython without the version. So it could not find the appropriate python library.

The referenced comment

edit

I've managed to get it working with the following configuration:

• extension-module enabled
• added the rustflags -C link-args=-L<LD_LIBRARY_PATH> -lpythonX.Ym

And it ONLY works with the m version of the library. I tested this for targets aarch64-linux-android and armv7-linux-androideabi.

Since this is a linking issue with extension-module would it be related to #1040 ?

[davidhewitt]

Thanks for the update! Perhaps not #1040 but definitely worth remembering in this area. Let's keep this issue open until we adjust build scripts accordingly.

To figure out what's going on here - so it sounds to me like libpython.so does not exist in the distribution you're compiling for, but presumably libpython3.8m.so does?

[davidhewitt]

(I'm a bit confused because I thought the last python I'd seen where the library had the m suffix was 3.6, but I've never found documentation about what the suffix was for, so perhaps there's something else at play here...)

[programmerjake]

(I'm a bit confused because I thought the last python I'd seen where the library had the m suffix was 3.6, but I've never found documentation about what the suffix was for, so perhaps there's something else at play here...)

my guess is m is for minimal like in the python3-minimal debian package.

[davidhewitt]

Aha looks like it's showing if the ABI has pymalloc support. We can check it with sys.abiflags. However, it was definitely removed in 3.8: https://docs.python.org/3/whatsnew/3.8.html#build-and-c-api-changes

We could fix build scripts to check this for Python 3.5-3.7.

@Progdrasil, you're definitely cross compiling against Python 3.8 when you have this m suffix?!

[Progdrasil]

Thats really interesting because i am compiling against 3.8. and both the 'm' version and normal versions are available. Ive tried linking against both and only the 'm' version works at runtime. If i link against the normal version i get the symbol not found error at runtime

I am curious to try compiling against 3.7 now. I'll update you when I have

[davidhewitt]

After exploring the python-for-android codebase it looks to me like they're generating this libpython3.8m.so name themselves, so I've opened an upstream patch.

There are probably still fixes we can introduce on Python 3.5-3.7 for this m suffix, where it is a valid identifier. EDIT: looks like we check sysconfig.get_config_var('LDVERSION') in build.rs, which (at least on my system) does include the m suffix for Python 3.7.

[Progdrasil]

looks like we check sysconfig.get_config_var('LDVERSION') in build.rs, which (at least on my system) does include the m suffix for Python 3.7.

Yes but that is used for local compilation, not cross compilation as I am doing. Instead you parse the header files and apply the major and minor version. Is there a way to get the configuration of the cross compiled version to see if it is being compiled as the m variant?

[davidhewitt]

Yes but that is used for local compilation, not cross compilation as I am doing. Instead you parse the header files and apply the major and minor version. Is there a way to get the configuration of the cross compiled version to see if it is being compiled as the m variant?

Indeed, you're correct and I was thinking about this this morning too. Actually I think that the cross compilation header parsing is both broken and brittle; I'm thinking about whether we need to try a solution using bindgen.

If I were to kick off a PR to use bindgen to detect cross-compile info, would you be willing to help test it? (And maybe even help finish it off, depending on how much time I can find for it...?)

[Progdrasil]

Also I succeeded in compiling for 3.7 and running the sysconfig.get_config_var(LDVERSION) on the target for 3.7 it returned 3.7m and when running with 3.8 it returned 3.8. However I think your merge request helps because the issue on the target was the fact that I was linking against libpython3.8.so while the only object that was copied to the target was libpython3.8m.so.

[Progdrasil]

Yeah I would be glad to help out! I can also help contribute to it as well

[davidhewitt]

Great! I'll try to either get notes or a rough implementation written up tomorrow or soon after.

[davidhewitt]

So I took a look at the bindgen approach this morning, but it looks like that's not the way to go about this either. A lot of the variables are set by the build system rather than configuration headers, so parsing the headers doesn't give us everything we want.

Instead, I started looking at the sysconfig module and saw that we might be able to take another approach. Instead of running the python interpreter to extract sysconfig, I think we might be able to parse the underlying data file. Some examples of the data file on my system:

/usr/lib/python3.8/_sysconfigdata__x86_64-linux-gnu.py
/usr/lib/python3.7/_sysconfigdata__m_linux_x86_64-linux-gnu.py

Also interesting to note that the name of the data file has that "m" abiflag again, so it might be possible to infer that from the name of the sysconfigdata script.

Afraid I've got a lot of other things to focus on at the moment, so I'm not sure I have time to write the implementation right now. However, I'd be very willing to help design and review the implmentation if you want to take this on?

If this is potentially nontrivial it could be worth factoring it out into a crate rather than inlining it all into pyo3's build.rs. I actually already started a crate which might be appropriate https://github.com/davidhewitt/python-config - perhaps if we can get a working draft of this in that crate then we can subsequently move this crate into the pyo3 organisation and update pyo3's build script to use it?

[Progdrasil]

Sure I can start on this in my free time, might be able to do a bit tomorrow evening

[davidhewitt]

👍 many thanks, and no rush at all. Any time you're willing to spend helping out is much appreciated, and I'll try to put some time in where I can! Hopefully over time we'll be able to put together a good cross-compiling support for pyo3 😄

Oh, also, while searching documentation this morning I came across another tidbit which would also contribute to the linking issues you're experiencing: #1082.

[Progdrasil]

I'll keep that in mind with the cross compilation.

[davidhewitt]

Closed by #1095. (Thanks very much again for your hard work!)