poetry should upgrade pip and setuptools in virtualenvs it creates

[cjw296]

• [X] I am on the latest Poetry version.

• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: RHEL7.5

• Poetry version: Poetry version 1.0.3

Issue

Poetry does not appear to upgrade the version of pip that appears in virtualenvs it creates and manages. This means that, in my case, I get an ancient pip 10.x that doesn't know to use binary wheels, causing installs of packages to fail as I don't have python-devel on the machines I deploy to.

When poetry creates a virtualenv, it should, as part of that process, upgrade pip to the latest version available, and continue to upgrade every time it's about to do anything that installs or removes packages from a virtualenv.

The workaround for me is:

python3.6 `which poetry` shell
pip install -U pip

[dbanty]

I think the default behavior is appropriate, pip has been known to make changes that break Poetry, so auto-updating seems dangerous. This is probably more of a feature request for maybe a config setting to auto-update?

As a very slightly easier command you can do poetry run pip install --upgrade pip

[cjw296]

I somewhat agree on the "may break stuff", but the version of pip that ships with, for example, Python 3.6 is dangerously old. The pip maintainers want everyone to be on the latest and greatest, for some very good reasons. I really don't think the default should be to use the ancient pip that likely comes with your python distribution.

[cjw296]

Just got bitten by this again today, poetry add-ing packages to build up a project, but when I came to actually use a CLI script in the project I got a selection of:

pkg_resources.DistributionNotFound: The 'pyjwt>=1.0.0; extra == "signedtoken"' distribution was not found and is required by oauthlib
pkg_resources.DistributionNotFound: The 'cryptography; extra == "signedtoken"' distribution was not found and is required by oauthlib
pkg_resources.DistributionNotFound: The 'pyOpenSSL>=0.14; extra == "secure"' distribution was not found and is required by urllib3

Quickly tiring of this whackamole, I eventually remembered that the venv poetry creates on the Python I'm using (3.6) will have the crazy old versions of pip and setuptools in them which needed upgrading:

$ poetry run pip install -U pip setuptools
...
Installing collected packages: pip, setuptools
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed pip-20.0.2 setuptools-45.2.0

I appreciate that the fail here is on the part of Python itself (shipping with ancient versions of those libraries with no way to upgrade them...) but it's pretty frustrating to have to remember to do poetry run pip install -U pip setuptools as part of setting up any poetry env.

[kawing-chiu]

I have an idea: maybe pip's version should be put into pyproject.toml, just like python's version:

[tool.poetry.dependencies]  
python = "^3.8"
pip = "^19.2"

It can be special-cased so that poetry update will not update it, just like python, unless poetry update pip is called. Initially, poetry can specify a version (range) that it thinks is safe for it.

Here is another example why this is useful: The latest PyQt5 release 5.14.1 switches to the manylinux2014 platform tag, so it's wheel can only be handled by a very recent pip (only pip versions >= 19.3 recognize this platform tag). If pip's version can be put into pyproject.toml, it looks like:

[tool.poetry.dependencies]  
python = "^3.8"
pip = "^19.3"
pyqt5 = "^5.14"

Then there is no need for any manual upgrade of pip when the project is installed.

[absassi]

@kawing-chiu Note that pip switched to calendar-based versioning in 2018, so version ranges cannot be set using semantic operators, such as ^. The Deprecation Policy says 6 months (unclear to me if that means 2 or 3 releases), so version ranges can still be set using < operator for next few upcoming releases.

[kawing-chiu]

@absassi Hi. I'm not sure I understand your point. Currently poetry just ignores the pip entry if you try to put one into pyproject.toml, so no matter what version you specify, it won't work. My proposal is that it should be picked up and handled by poetry. This is orthogonal to what pip version should be specified.

[absassi]

@kawing-chiu my point is that ^19.2 and ^19.3 in your examples are not good ranges for pip and I thought it was worth mentioning, in case the developers decide to accept it and include a default range in poetry like that. Other than this detail, I like your proposal.

[mblakesley]

Just ran into this myself. When I finally discovered this was the root cause of my problem, I got it working with poetry run pip install --upgrade pip. But I'm not happy with having to run that every time after poetry init. That's clunky. IMO poetry should have a way to do this "directly", not via poetry run pip ....

E.g. someone else suggested adding the feature as an option, like: poetry install --upgrade-pip-first (link: https://github.com/python-poetry/poetry/issues/1661#issue-531169149). That would work for me!

Note: I also tried poetry add pip which modifies the .toml file but doesn't actually upgrade pip. According to other comments on this project, poetry uses pip as a dependency so I can understand poetry not honoring that.

[finswimmer]

poetry now uses virtualenv to create a new venv. virtualenv ensures to get latest version of pip.

[adriantre]

@finswimmer as of which version? 1.1.4 still yields WARNING: You are using pip version 20.2.2; however, version 20.3.3 is available. for me.

[sinoroc]

As far as I understood _poetry_ does not intervene here. It is _virtualenv_ (a dependency of _poetry_) that takes care of installing _pip_ and _setuptools_ in the virtual environments. _virtualenv_ has its own schedule and rules to decide when to update its internal copies of _pip_ and _setuptools_ that will be used for subsequent creations of virtual environments. It is described in _virtualenv_'s user guide. Running virtualenv --upgrade-embed-wheels can enforce an update of _virtualenv_'s internal copies of _pip_ and _setuptools_.

This could probably be a FAQ. Or maybe this could be moved to a _Q&A_ discussion.