Setuptools: easy_install (setup.py install) doesn't ignore pre-releases

[astrofrog]

It looks like install_requires causes the latest version of a package, even pre-releases, to be installed. If I set up a simple setup.py file with Matplotlib as a dependency:

from setuptools import setup
setup(install_requires=['matplotlib'])

it will currently download and install 2.0.0b4:

$ python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to UNKNOWN.egg-info/requires.txt
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing UNKNOWN.egg-info/PKG-INFO
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build
creating build/bdist.macosx-10.6-x86_64
creating build/bdist.macosx-10.6-x86_64/egg
creating build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying UNKNOWN.egg-info/PKG-INFO -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying UNKNOWN.egg-info/SOURCES.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying UNKNOWN.egg-info/dependency_links.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying UNKNOWN.egg-info/requires.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying UNKNOWN.egg-info/top_level.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/UNKNOWN-0.0.0-py3.5.egg' and adding 'build/bdist.macosx-10.6-x86_64/egg' to it
removing 'build/bdist.macosx-10.6-x86_64/egg' (and everything under it)
Processing UNKNOWN-0.0.0-py3.5.egg
Copying UNKNOWN-0.0.0-py3.5.egg to /Users/tom/miniconda3/envs/empty/lib/python3.5/site-packages
Adding UNKNOWN 0.0.0 to easy-install.pth file

Installed /Users/tom/miniconda3/envs/empty/lib/python3.5/site-packages/UNKNOWN-0.0.0-py3.5.egg
Processing dependencies for UNKNOWN==0.0.0
Searching for matplotlib
Reading https://pypi.python.org/simple/matplotlib/
Downloading https://pypi.python.org/packages/36/01/8fc19448d018b5a58cf658a4521fa65026e1110816d90eb79144e3ee75e9/matplotlib-2.0.0b4.tar.gz#md5=250b086c52b89844ef61dd5ed0b398bb

Ideally, this should behave like pip and only pick the latest real release (pip has a --pre flag to select pre-releases).

[jaraco]

You're right. easy_install has never had support for excluding pre-releases other than by excluding a range that includes pre-releases (e.g. matplotlib<2.0.0dev or matplotlib>=2.0.0).

Your best bet is to use pip to install your packages (i.e. pip install .).

I would be interested in a patch that harmonizes the behavior between pip install and easy_install, perhaps even having easy_install simply invoke pip under the hood. I don't think the community has the desire to re-implement the features of pip within easy_install, but I'm also open to modest contributions in that direction.

[leorochael]

Using pip to install packages from easy_install() has two major requirements/consequences:

• All options of easy_install() should map somehow to command line switches of pip, enough so that consumers of easy_install() (like buildout) keep working

  • Or to a combination of pip switches plus manual adjustments (e.g. script installation, .pth construction, etc.)

• Making pip a (vendored?) dependency of setuptools

An alternative route, which I believe would be less painful for all parties involved, is to lift functionality from pip into packaging (which is already vendored into both pip and setuptools) so that the install functionality of both pip and easy_install() becomes a simple shim or pilot on top of a packaging API for package installation.

This would also help with things like having easy_install() able to install wheels.

[dstufft]

An alternative route, which I believe would be less painful for all parties involved, is to lift functionality from pip into packaging (which is already vendored into both pip and setuptools) so that the install functionality of both pip and easy_install() becomes a simple shim or pilot on top of a packaging API for package installation.

This is effectively already happening although packaging is a "some assembly required" kind of library. In this particular case when I switched setuptools over to using packaging I hardcoded a prereleases=True because at the time I didn't feel like figuring out how to plumb a --pre flag through all the places that needed it. AFAIK setuptools already has all of the things it needs to do this, it just needs someone to do the integration work to make it happen.

[mltsy]

Oh my! It seems pretty important to avoid installing pre-release builds! https://www.python.org/dev/peps/pep-0440/#handling-of-pre-releases

I'm very new to python, so I'm not in a position to contribute code unfortunately... but it seems like based on the PEP above, it might even be sufficient (for now) to simply turn off pre-release installation (and let it only use pre-releases that are already installed, or in cases where no other release matches the requirements). Having pre-releases turned on by default seems like a worse option than not being able to install pre-releases at all. What is the rationale for having it turned on by default?

[jaraco]

What is the rationale for having it turned on by default?

Easy-install was implemented long before pep 440, back to around 2005. Back then, the version handling was more simple, with basic rules around version parsing and ordering. Easy-install would install what you asked for within the ranges you specified. There wasn't another 'pre-release' dimension (except that dev, a, b were considered less than 0).

I don't think there's value at this point in implementing this behavior. Installation of packages to satisfy setup_requires is now backed by pip and easy-install is deprecated. Just use pip, which honors the pep.

[mltsy]

Ah - Thanks for the explanation! I'm working in an older project and I was not aware that setup_requires was the new way to specify dependencies. (?) (Probably due to my newness to python). I just looked up that option here: https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords

From what I read, it sounds like I would still need to use install_requires in order to install the packages (setup_requires just downloads them) - is that right? In which case, I don't see how the problem is solved... can you clarify what you mean when you say setup_requires is "backed by pip" and how you recommend specifying and installing dependencies with setuptools in a way that avoids pre-release versions being installed?

(I'm asking because setuptools just started installing a pre-release version of grpcio that is killing our build time)

[jaraco]

Sorry for the confusion. Let me try to clarify.

• install_requires is what's required for your package to run.
• setup_requires is what's required to build your package (when built by setuptools).

Easy_install is setuptools' built-in installer and it's been superseded by pip. You should always use pip to install your package (never run setup.py install or easy_install <anything>). Instead, pip install everything. If you use pip to install things, especially if those projects use a pyproject.toml to declare their build dependencies or if you have the build dependencies installed in advance, you shouldn't find that setuptools is installing anything and you shouldn't have to worry about pre-releases.

[mltsy]

Ah - thank you again! 🙏 I think I understand. Just to confirm though (and for posterity) - the most basic solution you are recommending is: keep setup.py as is, but instead of running python setup.py install just run pip install .?

To take it further, you mentioned that it may be useful to define some dependencies more explicitly in a pyproject.toml file (as per PEP 518 - although where to define which dependencies seems to be the topic of much debate).

Finally, although you did not mention this, I think it's worth noting (and confirming) that the Python tutorial now recommends using pipenv and a Pipfile for managing application dependencies, in cases such as deployable/web applications: https://packaging.python.org/tutorials/managing-dependencies/

This is as opposed to projects that are to be distributed as a package, which should continue to define their dependencies as described above, using setuptools' install_requires and setup_requires features: https://packaging.python.org/guides/distributing-packages-using-setuptools/

Does that all seem accurate?

[jaraco]

instead of running python setup.py install just run pip install .?

Yes!

Does that all seem accurate?

Yes. I personally don't distinguish between a deployable application and a library, so I use setuptools even for my own personal website, and so pipenv is a bad fit for my workflows. That said, it's perfectly reasonable if not preferable for someone developing an application to use a separate technique such as pipenv.