Setuptools: cython in setup_requires doesn't work

[louisabraham]

It seems I have an issue similar to #288

I would like to use a Cython extension without providing a .c file and without ensuring Cython is installed before.

According to https://setuptools.readthedocs.io/en/latest/history.html#id302 and https://stackoverflow.com/a/38057196/5133167 it should be possible.

I get

error: unknown file type '.pyx'

If I install cython, then it works. Thus, the behaviour of setup.py is not the same if cython was installed before or is installed using setup_requires.

[jaraco]

Sorry to hear that. You should be able to include setup_requires=['cython'], in your setup.py and it should work. Did you try that?

[louisabraham]

I did that. "the behaviour of setup.py is not the same if cython was installed before or is installed using setup_requires"

I think the problem is that Extension is evaluated before any setup_requires installation.

[pganssle]

Yeah, I've encountered this before, but I don't know that there's an easy solution to it. The extension and the setup_requires both need to be passed to the setup function, so Extension needs to be pulled in before setup is executed.

In Rust extensions, I've seen this done, which basically solves the problem.

Another possible option would be to add some mechanism for deferred execution, so the import statements can be nested in commands run only after setup_requires has installed the missing modules.

I was under the impression that the one true solution to this is PEP 518, and setup-time requirements will be resolved from pyproject.toml.

[louisabraham]

Nice one for rust. I think that's the best thing to do. A simple wrapper would call this for all the setup_requires. The only problem is that it would also mess with the installed package (the setup_requires are not installed in sites-packages but in a temp folder instead), so I guess doing it like that is explicit and simple enough.

[jaraco]

Indeed, the setup_requires options aren't available until after setup is executed, but I thought that the Extensions could be defined in such a way that they weren't evaluated until after setup_requires was evaluated. I went to a lot of work to get this all to work, and I'm pretty sure I tested it when I worked through the issue. Do you have a simple example that doesn't work?

I created this minimal setup script with an empty foo.pyx and it runs with setup.py build without error:

import setuptools

setuptools.setup(
    name='foo',
    setup_requires=['cython'],
    ext_modules=[
        setuptools.Extension(name='foo', sources=['foo.pyx']),
    ],
)

Can you share an example of what doesn't work (preferably a minimal example)?

[pganssle]

In the past I've had problems with the Cython documentation examples, which use cythonize, because they tend to use from Cython.build import cythonize

That also includes this example:

setup(
    setup_requires=[
        'cython>=0.x',
    ],
    extensions = [Extension("*", ["*.pyx"])],
    cmdclass={'build_ext': Cython.Build.build_ext},
    ...
)

I'm not quite sure how that ever worked without cython already installed on the system. If Cython is special-cased in this way, maybe it's worth making a PR or issue against the documentation. I'm not sure that helps the rust situation I mentioned, but I guess pyproject.toml will solve that eventually.

[pganssle]

If we did want to support this use case, we could create a MagicMock-like class that defers imports in the case of missing modules until after setup_requires has fired off. Something like:

from setuptools import deferred_import
Cython = deferred_import('Cython')
cythonize = Cython.cythonize

Edit Thinking about this - this would probably be very complicated to do correctly. Probably better to just have a separate mechanism for firing off setup_requires before setup is called.

[louisabraham]

@jaraco I can reproduce your behavior, on your example. I originally got the bug from https://github.com/bootphon/ABXpy/issues/11, I will work on a minimal example.

[jkp]

Just to add another voice - this issue is affecting me as well: I'll subscribe and watch for updates.

[jaraco]

Probably better to just have a separate mechanism for firing off setup_requires before setup is called.

Indeed, and PEP518 does just that (via pyproject.toml). It does require pip 10. It obviates setup_requires, but only for environments where pip 10 can be relied upon. I figure about 18 months before it will be generally available. I had hoped the setup_requires functionality would be a useful stopgap, but for cases where it's not viable, I think the only option is to wait for pip 10 to become dominant (and of course validate/test those concepts in the meantime).

I don't think there's anything left that setuptools can do here, so I'm closing the ticket, but I'm not seeking to stifle discussion. If there's something setuptools can or should do here, we can re-open and explore.