Pip-tools: pip-sync on fresh conda environment with Python 2.7.15 tries to remove Python package

[sp-bot-tools]

When running pip-sync in a conda environment with python 2.7.15 installed package, it always tries to uninstall non-pip packages like python, failing to execute.
It runs ok on conda environments with python 3.x.

Environment Versions

1. OS Type: MacOSX 10.13.6
2. Python version: Python 2.7.15 :: Anaconda, Inc.
3. pip version: pip 18.1 from /anaconda2/envs/my-env/lib/python2.7/site-packages/pip (python 2.7)
4. pip-tools version: pip-compile, version 3.1.0

Steps to replicate

1. conda create -n my-env python=2.7
2. source activate my-env
3. pip install -U pip
4. pip install pip-tools
5. echo "django" > requirements.in
6. pip-compile requirements.in
7. pip-sync requirements.txt

Expected result

pip-sync execution successful. A clean environment with just Django installed to the latest possible version.

Actual result

pip-sync error:

Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. Traceback (most recent call last): File "/anaconda2/envs/my-env/bin/pip-sync", line 11, in <module> sys.exit(cli()) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli install_flags=install_flags)) File "/anaconda2/envs/my-env/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall)) File "/anaconda2/envs/my-env/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/anaconda2/envs/my-env/bin/pip', 'uninstall', '-y', u'certifi', u'python', u'wsgiref']' returned non-zero exit status 1

[sp-ricard-valverde]

Sorry, it was me the one from the previous post. Didn't notice I was logged in as one of our bot accounts 😝

[peterHoburg]

I am having the same issue.

I have tried this with fresh conda environments, old ones, simple requirements.txt or complicated ones. All of them fail.

1. OS Type: MacOSX 10.14
2. Python Version: Python 2.7.14 |Anaconda, Inc.
3. pip version: pip 10.0.1 from /Users/username_here/miniconda3/envs/api/lib/python2.7/site-packages/pip (python 2.7)
4. pip-tools version: pip-tools==3.1.0

Steps to Replicate:

1. Make a fresh conda environment with conda create --name example python=2.7.14 pip
2. pip install pip-tools
3. navigate to my directory with a requirements.txt in it
4. pip-sync

Expected result:
Install all of the python packages in requirements.txt

Actual results:

$ pip-sync
Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Users/peterhoburg/miniconda3/envs/api/bin/pip-sync", line 11, in <module>
    sys.exit(cli())
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli
    install_flags=install_flags))
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync
    check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall))
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/peterhoburg/miniconda3/envs/api/bin/pip', 'uninstall', '-y', u'python', u'wsgiref']' returned non-zero exit status 1

[gsong]

This is reproducible in a generic Python 2.7 Docker container as well. Rolling back to pip 9.x is a work-around:

1. Start a Python 2.7 Docker container

   docker run --rm -it python:2.7 /bin/bash
   

2. Try to install requirements using stock pip

   $ pip --version
   pip 18.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)
   
   $ cd /tmp
   
   $ echo "requests" > requirements.in
   
   $ pip install pip-tools
   Collecting pip-tools
     Downloading https://files.pythonhosted.org/packages/4a/16/7d7e908a657b8b41ce665e38caaa1a4415707bdb7b06524deec9802f0b1c/pip_tools-3.2.0-py2.py3-none-any.whl (43kB)
       100% |████████████████████████████████| 51kB 3.4MB/s
   Collecting click>=6 (from pip-tools)
     Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
       100% |████████████████████████████████| 81kB 6.9MB/s
   Collecting six (from pip-tools)
     Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
   Installing collected packages: click, six, pip-tools
   Successfully installed click-7.0 pip-tools-3.2.0 six-1.12.0
   
   $ pip-compile
   #
   # This file is autogenerated by pip-compile
   # To update, run:
   #
   #    pip-compile --output-file requirements.txt requirements.in
   #
   certifi==2018.11.29       # via requests
   chardet==3.0.4            # via requests
   idna==2.8                 # via requests
   requests==2.21.0
   urllib3==1.24.1           # via requests
   
   $ pip-sync
   Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
   Traceback (most recent call last):
     File "/usr/local/bin/pip-sync", line 11, in <module>
       sys.exit(cli())
     File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
       return self.main(*args, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
       rv = self.invoke(ctx)
     File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
       return ctx.invoke(self.callback, **ctx.params)
     File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
       return callback(*args, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli
       install_flags=install_flags))
     File "/usr/local/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync
       check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall))
     File "/usr/local/lib/python2.7/subprocess.py", line 190, in check_call
       raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['/usr/local/bin/pip', 'uninstall', '-y', u'python', u'virtualenv', u'wsgiref']' returned non-zero exit status 1
   

3. Roll back to pip 9.x

   $ pip install "pip<10"
   Collecting pip<10
     Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
       100% |████████████████████████████████| 1.4MB 8.5MB/s
   Installing collected packages: pip
     Found existing installation: pip 18.1
       Uninstalling pip-18.1:
         Successfully uninstalled pip-18.1
   Successfully installed pip-9.0.3
   
   $ pip-sync
   DEPRECATION: Uninstalling a distutils installed project (python) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
   Uninstalling Python-2.7.15:
     Successfully uninstalled Python-2.7.15
   Uninstalling virtualenv-16.1.0:
     Successfully uninstalled virtualenv-16.1.0
   Not uninstalling wsgiref at /usr/local/lib/python2.7, as it is in the standard library.
   You are using pip version 9.0.3, however version 18.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   Collecting certifi==2018.11.29
     Cache entry deserialization failed, entry ignored
     Cache entry deserialization failed, entry ignored
     Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
       100% |████████████████████████████████| 163kB 4.6MB/s
   Collecting chardet==3.0.4
     Cache entry deserialization failed, entry ignored
     Cache entry deserialization failed, entry ignored
     Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
       100% |████████████████████████████████| 143kB 5.3MB/s
   Collecting idna==2.8
     Cache entry deserialization failed, entry ignored
     Cache entry deserialization failed, entry ignored
     Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
       100% |████████████████████████████████| 61kB 5.0MB/s
   Collecting requests==2.21.0
     Cache entry deserialization failed, entry ignored
     Cache entry deserialization failed, entry ignored
     Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
       100% |████████████████████████████████| 61kB 10.0MB/s
   Collecting urllib3==1.24.1
     Cache entry deserialization failed, entry ignored
     Cache entry deserialization failed, entry ignored
     Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
       100% |████████████████████████████████| 122kB 6.4MB/s
   Installing collected packages: certifi, chardet, idna, urllib3, requests
   Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
   You are using pip version 9.0.3, however version 18.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   

[atugushev]

Thanks for the reports!

I've prepared a PR to fix this issue. Anyways, there is a workaround: pip-sync --user.