Cython: Simplify compilation of complete applications

[scoder]

It can be useful in some cases to compile a complete application into a single binary. Currently, this is difficult to do with Cython and involves several manual steps. Since most of this can be automated, there should be a tool for it.

A part of this is already implemented in the form of the cython_freeze tool and the BuildExecutable helper. Stitching it together into an easy to use "here's my code, drop the binary here" tool should not be all too difficult. Also take a look at Nuitka, which reportedly has a good way of dealing with this.

Also see #2849 regarding the integration of cython_freeze into cython --embed.

[jdodds]

I've run into two major issues when trying to use Cython to take a python codebase and create a single binary, some of which I'm sure are due to gross ignorance on my part:

1. As far as I know, there isn't such a thing as a package-aware c extension in python, so if you're, for example, running cython_freeze over the codebase of an application that contains multiple packages you need to include a step where you flatten that namespace. This isn't too hard to do in codebases that don't do much in __init__.py files for example, but is much more difficult to guarantee in the general case.

2. Related to the above, it's difficult to account for third-party dependencies. They still need, in general, to be shipped alongside the executable. Py2exe has an extension that allows for python to load dynamic libs from a zipfile, and also the capability to pack that up into a single executable, I'm not aware of a working way to do that on systems other than windows.

[ha11owed]

+1

I currently have the issue of not being able to define packages with PyImport_ExtendInittab.
With better support for this, it would not be necessary to flatten the namespaces.

[jdodds]

@ha11owed that's something that's a limitation of Python's C API and is out of Cython's hands as far as I'm aware. If it's possible to create a Python C extension that is seen as a package while being built into the interpreter I'd be glad to put in work getting that into Cython.

[scoder]

Copying the PyInstaller+Cython tutorial link from #2147 here for future reference:
https://github.com/mobiusklein/cython_pyinstaller_example

[joukewitteveen]

Let's get back to this issue with more concrete details.

@ha11owed that's something that's a limitation of Python's C API and is out of Cython's hands as far as I'm aware. If it's possible to create a Python C extension that is seen as a package while being built into the interpreter I'd be glad to put in work getting that into Cython.

In PEP489, I read that a compiled module pack.mod.sub should be in pack/mod/sub.so with an initialization function PyInit_sub. When embedding modules, we want them to become 'built-in' modules, which we achieve through PyImport_ExtendInittab (as done in cython_freeze).

This is where @jdodds is not entirely correct, as these built-in modules need not live at the toplevel (they can form a hierarchy). However, when using the default initialization function names (PyInit_<modulename>), there will be a naming collision between the initialization functions of pack.mod.sub and other.sub. Both functions will be named PyInit_sub.

It is possible to solve this without really breaking anything. For this, we need Cython/Compiler/ModuleNode.py:mod_init_func_cname to output the api_name instead when aiming for built-in modules. The inittab should then simply refer to these PyInit_<api_name> functions instead. The point is that we can use any function name we want in the inittab and this should somehow become unique for every module.

What is left, then, is to decide how to expose this functionality. With --embed-modules= of #2849, we can use PyInit_<api_name>, bun this means that we can only link to libraries built specifically for this purpose. Maybe something like cython --built-in pack/mod/sub.py should generate code with the API-name as the name of the initialization function.