Poetry: Poetry export does not include setuptools even when it is a direct dependency.

[apollo13]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 31
• Poetry version: 1.0.0b4 (at the time of writing b5 is not on PyPi, will retry later on)
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/apollo13/1d0c72978531d3a5688f1e3847fa6fe6

Issue

Using the pyproject.toml file from the Gist and executing poetry lock I get the following lockfile:

[[package]]
category = "main"
description = "WSGI HTTP Server for UNIX"
name = "gunicorn"
optional = false
python-versions = ">=3.4"
version = "20.0.0"

[package.dependencies]
setuptools = ">=3.0"

[package.extras]
eventlet = ["eventlet (>=0.9.7)"]
gevent = ["gevent (>=0.13)"]
setproctitle = ["setproctitle"]
tornado = ["tornado (>=0.2)"]

[metadata]
content-hash = "20c8a1cc7d149b3ad58ca5063dfd37654f03ba65ee8b484d2faefe7ce28b7daf"
python-versions = "^3.7"

[metadata.files]
gunicorn = [
    {file = "gunicorn-20.0.0-py2.py3-none-any.whl", hash = "sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb"},
    {file = "gunicorn-20.0.0.tar.gz", hash = "sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50"},
]

setuptools is correctly discovered as dependency of gunicorn (see https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77 -- it does indeed require it), but the lockfile does not contain any hashes for setuptools.
If I now export to requirements.txt the file looks like this:

gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50

Which is not installable with pip install --require-hashes --force-reinstall -r requirements.txt (note the --require-hases there):

Collecting gunicorn==20.0.0 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/60/0d/3dbda0324f5bf007f3274e5ea09f0f3bcbf0ca01a75b80ff4f1ff9f8ecfd/gunicorn-20.0.0-py2.py3-none-any.whl
Collecting setuptools>=3.0 (from gunicorn==20.0.0->-r requirements.txt (line 1))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=3.0 from https://files.pythonhosted.org/packages/d9/de/554b6310ac87c5b921bc45634b07b11394fe63bc4cb5176f5240addf18ab/setuptools-41.6.0-py2.py3-none-any.whl#sha256=3e8e8505e563631e7cb110d9ad82d135ee866b8146d5efe06e42be07a72db20a (from gunicorn==20.0.0->-r requirements.txt (line 1))
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

If I instead add Django via poetry add Django, the poetry.lock file will contain Django's dependencies (pytz and sqlparse):

pytz = [
    {file = "pytz-2019.3-py2.py3-none-any.whl", hash = "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d"},
    {file = "pytz-2019.3.tar.gz", hash = "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"},
]
sqlparse = [
    {file = "sqlparse-0.3.0-py2.py3-none-any.whl", hash = "sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177"},
    {file = "sqlparse-0.3.0.tar.gz", hash = "sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873"},
]

and also export them to the requirements file:

django==2.2.7 \
    --hash=sha256:89c2007ca4fa5b351a51a279eccff298520783b713bf28efb89dfb81c80ea49b \
    --hash=sha256:16040e1288c6c9f68c6da2fe75ebde83c0a158f6f5d54f4c5177b0c1478c5b86
gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873

I fear there is some special casing going on for setuptools, it would be great if setuptools also ended up in the lock and requirement files if it is a direct dependency of on of the added packages.

[apollo13]

Verified that this issue also exists on beta 5

[mbelang]

I think I have a problem that might be related to this issue.

I have a service which includes a dependency that pins setuptools to 40.8.0 and doing poetry install fails the first time with:

[EnvCommandError]                                                                                                                                                                                                   
Command ['/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/bin/python', '-m', 'pip', 'install', '-e', '/home/circleci/project'] errored with the following return code 2, and output:                                     
Looking in indexes: https://pypi.org/simple, ***************************************************                                                                                                                                 
Obtaining file:///home/circleci/project                                                                                                                                                                                          
ERROR: Exception:                                                                                                                                                                                                                
Traceback (most recent call last):                                                                                                                                                                                               
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 407, in check_if_exists                                                                
    self.satisfied_by = pkg_resources.get_distribution(str(no_marker))                                                                                                                                                           
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution                                                          
    dist = get_provider(dist)                                                                                                                                                                                                    
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider                                                              
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]                                                                                                                                                         
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require                                                                   
    needed = self.resolve(parse_requirements(requirements))                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve                                                                   
    raise VersionConflict(dist, req).with_context(dependent_req)                                                                                                                                                                 
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})  
During handling of the above exception, another exception occurred:                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                               
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main                                                                          
    status = self.run(options, args)                                                                                                                                                                                             
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run                                                                           
    resolver.resolve(requirement_set)                                                                                                                                                                                            
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve                                                                         
    self._resolve_one(requirement_set, req)                                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one                                                                    
    abstract_dist = self._get_abstract_dist_for(req_to_install)                                                                                                                                                                  
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 291, in _get_abstract_dist_for                                                          
    req, self.require_hashes, self.use_user_site, self.finder,                                                                                                                                                                   
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 255, in prepare_editable_requirement                                                
    req.check_if_exists(use_user_site)                                                                                                                                                                                           
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 418, in check_if_exists                                                                
    self.req.name                                                                                                                                                                                                                
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution                                                          
    dist = get_provider(dist)                                                                                                                                                                                                    
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider                                                              
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]                                                                                                                                                         
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require                                                                   
    needed = self.resolve(parse_requirements(requirements))                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve                                                                   
    raise VersionConflict(dist, req).with_context(dependent_req)                                                                                                                                                                 
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})  
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.                                                                                                                                                  
You should consider upgrading via the 'pip install --upgrade pip' command.                                                                                                                                                       

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

Exited with code 1

With the following pyproject.toml

[tool.poetry]
name = "usher-events"
version = "0.0.0"
description = "Blahblah"
authors = ["Blabla <[email protected]>"]
packages = [{ include = "usher_events" }]

[tool.poetry.dependencies]
python = "^3.7"
aws-xray-sdk = "^2.4.2"
botocore = "^1.12.237"
colorama = "^0.4.1"
datadog_lambda = "^0.6.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

If I do poetry install a second time it works without issue. The problem is that on CI it fails every time as CI is always starting fresh. I have poetry 0.12.17 and python 3.7.4.

[ryancausey]

Can confirm that setuptools as a dependency does not appear to be handled properly. There's a package that depends on setuptools in my lockfile, but there is not associated entry in the lockfile for setuptools itself.

Lockfile snippet:

[[package]]
category = "main"
description = "Library to implement a well-behaved Unix daemon process."
name = "python-daemon"
optional = false
python-versions = "*"
version = "2.2.4"

[package.dependencies]
docutils = "*"
lockfile = ">=0.10"
setuptools = "*"

[insysion]

This issue appears to affect pip as a package too.

[apollo13]

This:

diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):

 class Provider:

-    UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+    UNSAFE_PACKAGES = {}

     def __init__(self, package, pool, io):  # type: (Package, Pool, Any) -> None
         self._package = package

fixes my issue. Now the question is: Why are those unsafe and what are the consequences?

[insysion]

It seems there's not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)

At the very least, we should introduce an --allow-unsafe or similar flag, if not remove the unsafe packages hard-coding altogether.

[kminehart]

Also having this issue with 2 packages that require separate versions of setuptools. (pathvalidate and google-auth).

Install works the first time, but when install is ran again with a cached virtualenv (same lockfile, no changes to pyproject.toml):

[EnvCommandError]
Command ['/root/repo/.venv/bin/pip', 'install', '-e', '/root/repo'] errored with the following return code 2, and output: 
Obtaining file:///root/repo
Exception:
Traceback (most recent call last):
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1025, in check_if_exists
    self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 521, in _prepare_file
    req_to_install.check_if_exists()
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1036, in check_if_exists
    self.req.name
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})

I'm assuming this is what's happening:

• setuptools 40.3.0 is installed to satisfy the requirement for google-auth.
• setuptools 39.0.1 then is installed and satisfies the requirement for pathvalidate, overwriting the one installed for google-auth.

Once reran, the dependency already exists for google-auth, but it's not the right version, and thus causes the above error.

Just a guess, though.

[insysion]

I am considering opening a PR for this. I plan to change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.

[eliadl]

This:

diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):

 class Provider:

-    UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+    UNSAFE_PACKAGES = {}

     def __init__(self, package, pool, io):  # type: (Package, Pool, Any) -> None
         self._package = package

fixes my issue. Now the question is: Why are those unsafe and what are the consequences?

@sdispater ?

[gsemet]

Hello. Packaging setuptools may be needed for some reason poetry may not be aware of, so it is important to provide an option to not ignore it

Pinning it inside the virtualenv is also important. pipenv did the mistake of automatically installing the newer version of setuptools on new environment bootstrap, leaving it imposible to pin to a previous, working version in the lockfile (45.0.0 broke something, for a while it was impossible to go back to a version <45). So please allow pinning setuptools and other "special" packages (pip, poetry itself,...).

[sdispater]

The reason why they are marked as "unsafe" is because Poetry relies on them internally (well mostly pip).

However, Poetry will eventually no longer use them and when that happens this restriction will no longer be needed. Until then, I prefer that we leave it as it is to avoid unexpected side effects.

[insysion]

@sdispater, thanks for getting back to us. Did you see my proposal?

change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This would seem to address problem of locking a set of packages that are inconsistent due to the unsafe packages being completely ignored at resolve time without affecting the default install/export behaviour. Would you support a PR implementing this?

Sorry to push the subject but it's causing us significant pain right now: we're having to add dummy constraints updated manually to workaround this. It's a high maintenance and patchy state of affairs.

[ptramsey]

Any word on this? Just encountered this same issue; it'd be nice if there were a solution.

[ivallesp]

Same here. I found VersionConflict with the pip and setuptools versions when trying to install tensorflow and tensorboard from poetry. The only workaround I found was to substitute the poetry install by:

poetry run pip install -U pip
poetry run pip install -U setuptools
poetry install

This is not optimal as the pip and setuptools versions are not locked. Any solution to this? Seems to be a critical bug...

Issue #1651 seems to be tackling this point, although no activity since February

[vbersier]

Same situation here as @ivallesp , tensorboard requires setuptools >= 41.0.0 but for some reason, 40.8.0 gets installed. This gives me a ContextualVersionConflict runtime error from pkg_resources.

Even adding a fake dependency to my project for a more recent setuptools doesn't help, nothing gets installed when I run poetry.

[earonesty]

Possibly poetry just should install and manage its setuptools and pip requirements in different/edited/hacked sys.path (...venv/lib/poetry-packages/ instead of site-packages), and then the venv itself can have no restrictions. IE: poetry should never have 'unsafe' packages, because its environment is separately managed by itself. That way we don't have to wait until poetry no longer uses setuptools and/or pip. A side effect of this is that poetry can then bootstrap to use any support packages it needs (arrow or other fun things), because they won't be a problem.

[apollo13]

@earonesty brings up an interesting point. I'd argue that poetry itself is usually installed globally or in another venv (ie via pipx) so it should not conflict with anything in the venv at all. Does that address some of your concerns @sdispater? I am asking because this issue is the final show stopper for us to embrace poetry fully. I understand the motivation behind your argument, but if poetry isn't in the managed venv anyways, there are far less chances of issues. On the other hand, everyone installing gunicorn or tensorflow currently has problems with poetry.

[alexchandel]

@insysion Did you open a PR?

This is a serious bug. It causes the installation of older, buggier versions of setuptools like in https://github.com/pypa/setuptools/issues/1963. There's no reason to break such projects.

[earonesty]

I'd be happy to try to open a PR if people like the idea of putting poetry deps in another folder (. poetry_deps ?)

[earonesty]

note: i suspect poetry's dependence on msgpack also causes problems for the same reason. there's no way to alter msgpack's version when poetry is running. ie: you need to be sure the msgpack you run is identical to poetry's or you will fail on machines, like windows, that lock shared libs while in use.