pip install -e does not install setup.py install_requires dependencies

[leezu]

• Pip version: 9.0.1
• Python version: 3.6.3
• Operating system: Gentoo Linux

Description:

I have a local package that specifies a number of dependencies via the install_requires argument in setup.py. Installing it with pip install --user -e . does not install these dependencies, whereas pip install --user . does.

What I've run:

Consider the case where one of the install_requires dependencies (here gitpython) is not present on the system. Below is the output of pip install in both cases. Note that gitpython is only collected and installed in the second case.

➜  package git:(master) ✗ pip install --user -e .
Obtaining file:///home/leonard/projects/package
Requirement already satisfied: mxnet in /home/leonard/software/mxnet/python (from leezu-package==0.1.dev0)
Requirement already satisfied: numpy in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: addict in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: numba in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pyyaml in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: spacy in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_sm in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_md in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: matplotlib in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: xarray in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: netCDF4 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: progressbar2 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: requests in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: graphviz in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: llvmlite in /usr/lib64/python3.6/site-packages (from numba->leezu-package==0.1.dev0)
Requirement already satisfied: murmurhash<0.27,>=0.26 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cymem<1.32,>=1.30 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: thinc<6.6.0,>=6.5.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pip<10.0.0,>=9.0.0 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: six in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pathlib in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ujson>=1.35 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: dill<0.3,>=0.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: regex<2017.12.1,>=2017.4.1 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: python-dateutil in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pytz in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: cycler>=0.10 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: python-utils>=2.1.0 in /home/leonard/.local/lib64/python3.6/site-packages (from progressbar2->leezu-package==0.1.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in /home/leonard/.local/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: wrapt in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cytoolz<0.9,>=0.8 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: termcolor in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: wcwidth in /usr/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: html5lib in /home/leonard/.local/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: toolz>=0.8.0 in /home/leonard/.local/lib64/python3.6/site-packages (from cytoolz<0.9,>=0.8->thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Installing collected packages: leezu-package
  Running setup.py develop for leezu-package
Successfully installed leezu-package

➜  package git:(master) ✗ pip install --user .                                                                                                                                <<<
Processing /home/leonard/projects/package
  Requirement already satisfied (use --upgrade to upgrade): leezu-package==0.1.dev0 from file:///home/leonard/projects/package in ./src
Requirement already satisfied: mxnet in /home/leonard/software/mxnet/python (from leezu-package==0.1.dev0)
Requirement already satisfied: numpy in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: addict in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: numba in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pyyaml in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: spacy in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_sm in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_md in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: matplotlib in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: xarray in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: netCDF4 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: progressbar2 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: traitlets>=5.0.0.dev in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: singleton-decorator in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: visdom in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Collecting gitpython (from leezu-package==0.1.dev0)
  Using cached GitPython-2.1.7-py2.py3-none-any.whl
Requirement already satisfied: requests in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: graphviz in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: llvmlite in /usr/lib64/python3.6/site-packages (from numba->leezu-package==0.1.dev0)
Requirement already satisfied: murmurhash<0.27,>=0.26 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cymem<1.32,>=1.30 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: thinc<6.6.0,>=6.5.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pip<10.0.0,>=9.0.0 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: six in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pathlib in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ujson>=1.35 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: dill<0.3,>=0.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: regex<2017.12.1,>=2017.4.1 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: python-dateutil in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pytz in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: cycler>=0.10 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: python-utils>=2.1.0 in /home/leonard/.local/lib64/python3.6/site-packages (from progressbar2->leezu-package==0.1.dev0)
Requirement already satisfied: ipython_genutils in /usr/lib64/python3.6/site-packages (from traitlets>=5.0.0.dev->leezu-package==0.1.dev0)
Requirement already satisfied: decorator in /home/leonard/.local/lib64/python3.6/site-packages (from traitlets>=5.0.0.dev->leezu-package==0.1.dev0)
Requirement already satisfied: pillow in /usr/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: tornado in /home/leonard/.local/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: pyzmq in /usr/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: torchfile in /home/leonard/.local/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: gitdb2>=2.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from gitpython->leezu-package==0.1.dev0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in /home/leonard/.local/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: wrapt in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cytoolz<0.9,>=0.8 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: termcolor in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: html5lib in /home/leonard/.local/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: wcwidth in /usr/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: olefile in /usr/lib64/python3.6/site-packages (from pillow->visdom->leezu-package==0.1.dev0)
Requirement already satisfied: smmap2>=2.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from gitdb2>=2.0.0->gitpython->leezu-package==0.1.dev0)
Requirement already satisfied: toolz>=0.8.0 in /home/leonard/.local/lib64/python3.6/site-packages (from cytoolz<0.9,>=0.8->thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Installing collected packages: gitpython
Successfully installed gitpython-2.1.7

[ghost]

@leezu We're short on time, so one thing that you could do is write a test that exposes this bug and submit it as a PR.

[leezu]

@xoviat Sure. What would be the best way for checking if e.g. gitpython appears in the pip install output? Is it in order to depend on an external package such as gitpython for the test case?

[ghost]

See, for example, here:

https://github.com/pypa/pip/blob/66c53519c2629ec784659fba85c5a84c88144663/tests/functional/test_install.py#L138

[ghost]

The --pre option apparently doesn't work with -e either!

[pradyunsg]

Hi @leezu!

Could you share the setup.py file of your package? I just want to make sure there's nothing fishy happening there?

[leezu]

Hey @pradyunsg @xoviat . I have investigated this issue a bit further and found it was related to an
project_name.egg-info folder next to the setup.py whose ./project_name.egg-info/requires.txt didn't contain the up to date requirments.

Which means this bug boils down to:

• pip install -e uses the project_name.egg-info folder in the directory of ./setup.py and just reads the (possibly outdated) ./project_name.egg-info/requires.txt.
• pip install correctly parses setup.py and updates ./src/project_name.egg-info/requires.txt (assuming a layout where the code is in ./src/project_name )

Furthermore I found renaming the outdated project_name.egg-info folder to e.g. test.egg-info will not stop pip install -e from reading the (now not only outdated but also not wrongly named) test.egg-info/requires.txt file.

The following setup.py is sufficient to reproduce the bug:

from setuptools import setup
setup(name='project', install_requires=['gitpython'])

with the following files present in a test.egg-info folder in the same directory: PKG-INFO, requires.txt.

Not sure if the usage of the wrongly named egg-info file constitutes a bug. However I would expect pip install -e to at least double check the requires.txt file it uses with the contents of setup.py?

[pradyunsg]

Why is there an outdated project_name.egg-info in .? I think it should be removed but, well, I'm curious on how you got that there.

[pradyunsg]

/note multiple egg-info

[leezu]

Well this is a project I'm continuously developing. So at some point I added a new requirement to setup.py. When updating the project folder (via git) on a remote host I would do pip install -e --user . expecting that pip would install all newly added requirements.

(which it does when working with pip install --user .. So one workaround is to run pip install --user . pip install --user -e . in sequence to make sure all dependencies are catched..)

[leezu]

Sorry, I checked again: So just having an outdated ./src/project_name.egg-info/requires.txt is actually no problem. pip install -e . seems to update that correctly.

So the bug here really boils down to pip install -e . getting confused by the leftover ./project_name.egg-info/requires.txt which was created before I switched my project layout from having all code in ./src to having all code in ./src/project_name as advocated by https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure

While this is arguably an edge case, I think pip install -e . shouldn't be confused by it and at least double check the requires.txt file matches the setup.py requirements.. Or it should warn the user about the presence of this "wrong" ./project_name.egg-info folder.

What do you think?

[pradyunsg]

Ohkay. Thanks for the clarification.

I think this is because you switched layouts after doing an editable install and this is frankly an edge case. Maybe pip should detect it and warn/abort; idk how hard that is and find little motivation to do it myself. A PR for that would definitely be welcome though. :)

[pradyunsg]

If anyone wants to see this through, they are more than welcome to. What she'll/he'll probably want to do is look into how pip handles editable installs and see if it's possible to detect cases like this.

[pradyunsg]

Oh, btw, @leezu I think you should just delete the outdated egg-info directory in .. That'll resolve your issue. :)

[leezu]

@pradyunsg thanks. In case you're familiar with that part of the codebase (i.e. for the egg-info discovery) it would be great if you can post a link here.

[ghost]

I'd rather wait on this as it conflicts with a PR that I'm working on. I can fix it later.

[leezu]

Thanks @xoviat . For reference, I assume you refer to #4589 .

[ghost]

Yes. It's a large patch and so I would like to see it merged soon. But there are two problems:

1. I implemented the setuptools backend incorrectly the first time. Not a surprise because no one reviewed it (the setuptools maintainer accepted the PR after putting it off for three months). I guess that's better than the alternative of putting it off for longer though because at least I have a fix that will hopefully be in the next setuptools release (which means that the PR is delayed until them).
2. No one with write access has time (wants to) review #4589. I can't blame them because it's quite scary to change the functionality that I'm working on. But it's the only way that it will get done.

[pfmoore]

As a note, it's currently impossible to review #4589 as you're regularly adding new commits to it. I'd prefer to wait until it's stabilised and you're happy that the work is complete to your knowledge before trying to review. (I can't promise to be able to do a full review even then, as it's not a part of the code I know well, but I will at least take a look once the code settles down).

[ghost]

Thanks. Unfortunately, as I noted in the comment above, that will be after the next setuptools release.

[peterbe]

I stumbled into this too today. Using the latest pip 19.0.3 on Python 3.7.

I edited the list of install_requires in setup.py to add some-new-package and re-ran:

pip install -e .

But the new package wasn't installed. :(

The package was mentioned in inside remote_settings_uptake_health.egg-info/requires.txt but the comments above got me wondering if the reason was because I had two *.egg-info/ directories. Indeed I had.

The name key in setup(name=THIS, ...) had indeed changed and I suspect pip incorrectly looked at old_name.egg-info/requires.txt.

So I think the steps to reproduce are as follows:

1. Create a new package with just a setup.py file:

   ▶ cat setup.py
   from setuptools import setup

   setup(name="name-one", install_requires=["click"])

2. Install things:

▶ pip install -e .
Obtaining file:///Users/peterbe/dev/PYTHON/reproduce-pypa-issue-4780
Collecting click (from name-one==0.0.0)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Installing collected packages: click, name-one
  Running setup.py develop for name-one
Successfully installed click-7.0 name-one

1. Change your mind about the name and the list of requires:

▶ cat setup.py
from setuptools import setup

setup(name="name-one-v2", install_requires=["click", "requests"])

1. Try to install again:

▶ pip install -e .
Obtaining file:///Users/peterbe/dev/PYTHON/reproduce-pypa-issue-4780
Requirement already satisfied: click in /Users/peterbe/virtualenvs/reproduce-pypa-issue-4780/lib/python3.7/site-packages (from name-one==0.0.0) (7.0)
Installing collected packages: name-one
  Found existing installation: name-one 0.0.0
    Uninstalling name-one-0.0.0:
      Successfully uninstalled name-one-0.0.0
  Running setup.py develop for name-one
Successfully installed name-one

No requests installed this time!! Also, not the console output mentions "Installing collected packages: name-one" but that's not the name in my setup.py.

(Note; I edited the comment mentioning that the reason why it picks different *.egg-info/requires.txt might be related to the alphabetical order. I don't think that was right.)

[achimnol]

This is still an issue. In my projects, after updating setup.cfg (which has the install_requires mapping) and reinstalling the editable package does not update *.egg-info/requires.txt, breaking my plugin system based on pkg_resources.iter_entry_points() because it validates the package dependencies (just like pip check).

I need either a way to skip dependency validation when searching entry points or a way to ensure the latest *.egg-info/requires.txt content with editable installations.

[uranusjr]

I wonder if this is the problem I described in https://github.com/pypa/pip/pull/9147#discussion_r527750168? If editable requirements are always re-installed, setuptools would have a chance to re-build the egg-info directory when pip install -e is re-run.

Note that this may or may not be what we want, however, without PEP 517 standardising editable and incremental builds. Running setup.py can have a significant performance impact (even only for egg_info), and I can easily imagine projects requiring binary compilation may be relying on the current behaviour in their development workflow.

[achimnol]

@uranusjr is probably right.
Another problem I've encountered is that the first run of pip install -U -e . shows version conflict error but the second run does not, although pip check consistently reports version conflict errors (and pkg_resources.iter_entry_points() as well), after updating the requirements when there are incompatible requirements compared to the prior one.