Pip: When package is installed in editable mode from git, its build directory is not cleaned

[mitar]

Environment

• pip version:
• Python version:
• OS: Ubuntu 18.04

Description

When installing in editable mode from git, the build directory is not cleaned. This can lead to unnecessary disk consumption (in the example below, 123 MB).

Expected behavior

That build directory is removed unless --no-clean argument is used.

How to Reproduce

• pip3 install --upgrade pip
• pip3 install -e git+https://github.com/autonlab/fastlvm.git@f3229d5190a6eb5f0b3d20bb3e45f20f49feea92#egg=fastlvm
• du -sh /src/fastlvm/build

Output

root@696eeb452aee:/# pip3 install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 18.0MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.1.1
root@696eeb452aee:/# pip3 install -e git+https://github.com/autonlab/fastlvm.git@f3229d5190a6eb5f0b3d20bb3e45f20f49feea92#egg=fastlvm
Obtaining fastlvm from git+https://github.com/autonlab/fastlvm.git@f3229d5190a6eb5f0b3d20bb3e45f20f49feea92#egg=fastlvm
  Cloning https://github.com/autonlab/fastlvm.git (to revision f3229d5190a6eb5f0b3d20bb3e45f20f49feea92) to /src/fastlvm
  Running command git clone -q https://github.com/autonlab/fastlvm.git /src/fastlvm
Requirement already satisfied: numpy>=1.13.1 in /usr/local/lib/python3.6/dist-packages (from fastlvm) (1.16.3)
Requirement already satisfied: scipy>=0.17 in /usr/local/lib/python3.6/dist-packages (from fastlvm) (1.2.1)
Collecting scikit-learn>=0.18.1 (from fastlvm)
  Downloading https://files.pythonhosted.org/packages/90/c7/401c231c445fb6fad135e92197da9c3e77983de169ff1887cc18af94498d/scikit_learn-0.21.1-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)
     |████████████████████████████████| 6.7MB 2.6MB/s 
Collecting gensim (from fastlvm)
  Downloading https://files.pythonhosted.org/packages/d3/4b/19eecdf07d614665fa889857dc56ac965631c7bd816c3476d2f0cac6ea3b/gensim-3.7.3-cp36-cp36m-manylinux1_x86_64.whl (24.2MB)
     |████████████████████████████████| 24.2MB 9.0MB/s 
Collecting joblib>=0.11 (from scikit-learn>=0.18.1->fastlvm)
  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)
     |████████████████████████████████| 286kB 18.0MB/s 
Requirement already satisfied: six>=1.5.0 in /usr/lib/python3/dist-packages (from gensim->fastlvm) (1.11.0)
Collecting smart-open>=1.7.0 (from gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/bf/ba/7eaf3c0dbe601c43d88e449dcd7b61d385fe07c0167163f63f58ece7c1b5/smart_open-1.8.3.tar.gz (60kB)
     |████████████████████████████████| 61kB 19.1MB/s 
Collecting boto>=2.32 (from smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 36.4MB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.7.0->gensim->fastlvm) (2.21.0)
Collecting boto3 (from smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/4d/86/a94c87e948303b2c20c199ecc5d2780ada1889bb96234b1a5f4248d67e6a/boto3-1.9.153-py2.py3-none-any.whl (128kB)
     |████████████████████████████████| 133kB 32.9MB/s 
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.7.0->gensim->fastlvm) (3.0.4)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.7.0->gensim->fastlvm) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.7.0->gensim->fastlvm) (2019.3.9)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3/dist-packages (from requests->smart-open>=1.7.0->gensim->fastlvm) (2.6)
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3->smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/d7/de/5737f602e22073ecbded7a0c590707085e154e32b68d86545dcc31004c02/s3transfer-0.2.0-py2.py3-none-any.whl (69kB)
     |████████████████████████████████| 71kB 28.8MB/s 
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting botocore<1.13.0,>=1.12.153 (from boto3->smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/ec/3b/029218966ce62ae9824a18730de862ac8fc5a0e8083d07d1379815e7cca1/botocore-1.12.153-py2.py3-none-any.whl (5.4MB)
     |████████████████████████████████| 5.4MB 30.9MB/s 
Requirement already satisfied: docutils>=0.10 in /usr/lib/python3/dist-packages (from botocore<1.13.0,>=1.12.153->boto3->smart-open>=1.7.0->gensim->fastlvm) (0.14)
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.153->boto3->smart-open>=1.7.0->gensim->fastlvm)
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
     |████████████████████████████████| 235kB 27.8MB/s 
Building wheels for collected packages: smart-open
  Building wheel for smart-open (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/b8/cb/43/c0ba52baf2b0e371ec1d5b2d4685d6d24617b1391f3eeacda5
Successfully built smart-open
Installing collected packages: joblib, scikit-learn, boto, jmespath, python-dateutil, botocore, s3transfer, boto3, smart-open, gensim, fastlvm
  Running setup.py develop for fastlvm
Successfully installed boto-2.49.0 boto3-1.9.153 botocore-1.12.153 fastlvm gensim-3.7.3 jmespath-0.9.4 joblib-0.13.2 python-dateutil-2.8.0 s3transfer-0.2.0 scikit-learn-0.21.1 smart-open-1.8.3
root@696eeb452aee:/# du -sh /src/fastlvm/build
132M    /src/fastlvm/build

[chrahunt]

I'm not sure I understand the use case of an editable install if the build directory gets cleaned up by default, since rebuilds would take longer (especially for packages where the size matters, implying a lot of compilation was required). Is there any reason not to use pip install directly without the -e option?

[mitar]

-e provides access to git history, for example. Another use case is that provide access to test files, which through regular installation might not be available. So you might want to install and trigger testing, but not in all cases and not when publishing the package to PyPi. So -e mode provides some ways to get extra files during install, like whole repository.

[chrahunt]

That all seems very reasonable.

I don't know how strong of a justification is needed for changes in this area, so I'll leave that for someone else to comment on.

For editable installs (both VCS and for e.g. local directories) we run the equivalent of python setup.py develop, which leaves the build directory by default. We wouldn't want to assume a particular build directory, but can pass one explicitly with -b/--build-directory to the develop command and could clean it up afterwards.

It should be possible to do the same thing right now with e.g. tmpdir=$(mktemp -d) && pip install --install-option="--build-directory=$tmpdir" ... ; rm -rf $tmpdir, but for some reason setuptools still built into src/fastlvm/build when I tried locally.

I think it would be easier to get something like the above working with setuptools which could then be passed-thru using --global-option or --install-option than it would be to get tighter integration between pip and setuptools.

[chrahunt]

Of course it would also be possible to clone the repository explicitly to the /src directory independent of pip and then invoke the desired commands to install and clean the repository, but the point is probably to define all required dependency information in one place (e.g. requirements.txt) and have that behave in the expected way.