Pybind11: [QUESTION] How to build a pybind11 project to be independent from the Python interpreter on an end-user platform?

[Str4t0n]

How to build a pybind11 project to be independent from the Python interpreter on an end-user platform?

I have no problem building my project (since i have python installed on my PC), but as soon as I try to run the program on a different PC, it crases with "The program cannot be run, missing PythonXX.dll ...". I tried a workaround where I include the PythonXX.dll (from system32) with the program, but even with that, i get an error like "Py_Initialize: unable to load the file system codec. ImportError: No module named 'encodings'". I'm using a custom cmake file (because of work related stuff):

add_subdirectory(pybind11)
...
target_link_libraries( ... pybind11::embed)

there is no error while building, and there is also no error while running the program on my system.
For reference, i tried to use the original cmake too. (Because i thought the problem was in my custom cmake file)
I'm on windows, building a win32 c++ application (embedding), using Msbuild 15.9.21 and CL 19.16.27043, and Python34 (same error with Python37, but i don't think the version of Python matters in my case)
Might as well add that i don't want to create a .pyd library, I want an executable .exe that is able to use .py scripts inside it without the need to install Python on the end-user platform...

Is it even possible to do this?

[mrzealot]

Bumping this issue as I'm in the same situation. I thought linking against pybind11::embed would include the python lib into the eventual executable and that it would be independent of the system's python when it's run. But turns out that the same executable that can run fine on the system it was built on cannot run on another system... Apparently the python linking is dynamic? Is there a way to make it static and portable? Thanks a bunch!

[YannickJadoul]

This doesn't seem to be a particular pybind11 issue, though. Does this work when using the raw C API?

Next to that, have you checked the dependencies of the program (e.g. with Dependency Walker)? Maybe you can just ship PythonXY.dll along?

Does https://chris.hofstaedtler.name/blog/2013/01/embedding-python-on-win32.html help? It's an old article, but it seems to say it's not trivial to link statically, but it's possible?

[mrzealot]

@YannickJadoul thanks for your answer! Haven't tried the raw C API, but I don't see how that would be relevant. The pybind11 dependency is a given, so any solution would have to make that work independently of the system it was built on. This is also the problem with the article you linked: how does this relate to pybind11? (Tbf, I might not be seeing some obvious connection.)
However, I checked the verbose cmake output of my build (on linux), and supposedly it's linking against a certain libpython3.3m.a, which should be static by the looks of it.

Have also been talking with @Str4t0n in the background, awaiting dependency walker results. But apparently, his setup is linking against a python .lib, not a .dll, so the windows version should be static as well.

I guess I'm mostly just curious what the default, official stance is on pybind11 linking (as there's a big difference between us messing something up vs. pybind11 not being applicable or intended for this use-case in the first place).

[YannickJadoul]

@mrzealot

Haven't tried the raw C API, but I don't see how that would be relevant. The pybind11 dependency is a given, so any solution would have to make that work independently of the system it was built on.

Because you're linking against Python. pybind11 is header-only, so doesn't impose any extra runtime-dependencies. It's linking against Python that's problematic, and you would also get that if you use the C API rather than pybind11. So trying out the official example and seeing how things link together there would really help: https://docs.python.org/3/extending/embedding.html

This is also the problem with the article you linked: how does this relate to pybind11? (Tbf, I might not be seeing some obvious connection.)

So what I described above, that's the connection. All pybind11 does is provide a nice C++ interface on top of the C API. Nothing less, but definitely nothing more.

But apparently, his setup is linking against a python .lib, not a .dll, so the windows version should be static as well.

Not sure how Windows linking works, but I tend to remember that even with a DLL, you need a .lib file, containing the symbols that can later be found in the DLL? I might be wrong, though.

I guess I'm mostly just curious what the default, official stance is on pybind11 linking (as there's a big difference between us messing something up vs. pybind11 not being applicable or intended for this use-case in the first place).

There is none, as far as I'm aware. Again, when it comes to embedding, pybind11 provides a layer above the Python C API. That's why you're linking against things like libpython3.3m.a, which are the Python libraries, not pybind11. (Also, 3.3, really!? It's 2020; please get a newer, supported version of Python; who knows what bugs related to linking got solved meanwhile!)
And pybind11 also provides convenience CMake targets, of course. So if there is a way to statically link to Python, and there's a way pybind11 can provide a target for that (pybind11::embed_static or so), do please let us know.

[mrzealot]

@YannickJadoul You are absolutely right. My mistake was thinking that the pybind11::embed target was something special and not just a convenience (lack of sufficient familiarity with both pybind11 and cmake, I guess). Anyways, at the very least, be advised that the default pybind embedding is not necessarily static (definitely not on windows, probably not on linux either). But apart from this, I'll look into manual, non-convenience linking and see what I can achieve... Thanks!

[YannickJadoul]

@mrzealot, great! :-) I wish I could be of more help, but I'm not an expert on Windows development (except for knowing that this Dependency Walker program is often amazing and can show what's going wrong!), and I haven't used embedding in a real project yet.

Just curious: if Dependency Walker shows a dependency on PythonXY.dll, would you not be able to just ship that DLL along with the program?

Apart from that, I'll close this, as it's not a pybind11-specific issue, but please let us know if you find something interesting! It can be really useful for other users (even if the conclusion is that it's not possible). And please don't hesitate to reopen the issue if you find something and if pybind11 can do something (like providing another CMake target, or so) :-)

[mrzealot]

Hi @YannickJadoul , just wanted to quickly follow up on this. It is indeed possible to link statically against python, specifically we followed advice from this page. Do note, tho, that this only links with the .c/.h modules and the interpreter itself, but running programs also needs the standard lib, which still has to be available to the executable in either .py or .pyc format. But that (as opposed to the architecture-specific .dlls or .sos) can now be safely distributed along with the executable to achieve a full independence from python installations.

Re: how this relates to pybind11, I don't think there's any actual improvement necessary on the functionality side. Maybe a little more verbose explanation of how it all relates to python linking, and more emphasis on the fact that the pybind11::embed flag is just for convenience. I mean, I know (now) that the header-only nature of pybind is mentioned, but it definitely wouldn't hurt to reiterate for the embedding-specific sections, too. Also, a small (sub)section on static linking, mentioning the above resource and the fact that the standard lib is still going to be necessary would be a huge plus.

Anyway, thanks for your help, and hope this helps someone in the future...

[YannickJadoul]

Thanks, @mrzealot, for the update! Always good to know, indeed. Maybe we can see if we find a spot to add it to the docs, but it's quite hard to find where it best fits such that users will find it when needed?