Cython: Quick Test Routine After Import

[ax3l]

Hi,

we are currently facing an interesting issue downstream in setuptools with an imported, incompatible Cython install that might exist in a user's environment, e.g. while working with python virtual environments, conda or spack: https://github.com/pypa/setuptools/issues/1229#issuecomment-349644856

The main issue comes down to the fact that a downstream "optional usage of Cython" in the form of

try:
    # Attempt to use Cython for building extensions, if available
    from Cython.Distutils.build_ext import build_ext as _build_ext
except ImportError:
    # some fallback such as
    _build_ext = _du_build_ext

will not catch any errors that can occur when using a compiled library such as Cython. Exemplary errors can look like

failed to import Cython: /home/axel/.local/lib/python2.7/site-packages/Cython/Compiler/Scanning.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
Unexpected error: <class 'distutils.errors.DistutilsPlatformError'>

in the case of symbol mismatches between currently used virtual environment, system or user libraries. Unfortunately, such errors are not triggered on import but as soon as the imported functionality is used later on. (A full reproducer is posted here including a Dockerfile.)

We were wondering if there already is or if one could add a short, side-effect free "test" interface that one could call directly after the import in the same try scope in order to reliably trigger and catch incompatibly imported Cython installs?

Any feedback or suggestions are very welcome!

[scoder]

Three things:

• It's intentional that importing the build_ext module does not import the compiler.
• It would be better to import new_build_ext instead of build_ext. It's much closer to what people would expect as a default behaviour these days, since it calls cythonize() internally.
• You can just import the compiler explicitly to detect errors like the one you mentioned. Just import Cython.Compiler.Main or from Cython.Build import cythonize or something like that. Drawback is, obviously, that all of Cython gets imported early even if it's not used at all.

[ax3l]

@jaraco would one of the above solutions be suitable for you, downstream in setuptools?

[jaraco]

The third option looks like it may be the officially-supported way to detect if the compiler is available. Probably just __import__('Cython.Compiler.Main') with a comment explaining why.

[scoder]

Closing, as I think an explicit import solves this issue.