Pip-tools: pip-sync using global pip inside virtualenvs on Ubuntu

[alexwforsythe]

pip-sync seems to use the user-installed pip to detect installed distributions when invoked inside a virtualenv on Ubuntu (sorry in advance for the wall of text):

alex@spicylegato:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 15.10
Release:    15.10
Codename:   wily
alex@spicylegato:~$ pip2.7 --version
pip 8.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7)
alex@spicylegato:~$ sudo -H pip2.7 install pip-tools
Collecting pip-tools
  Using cached pip_tools-1.6-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Using cached click-6.3-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Installing collected packages: six, click, first, pip-tools
Successfully installed click-6.3 first-2.0.1 pip-tools-1.6 six-1.10.0
alex@spicylegato:~$ type -a pip-sync
pip-sync is /usr/local/bin/pip-sync
alex@spicylegato:~$ pip-sync --version
pip-sync, version 1.6
alex@spicylegato:~$ pip-compile requirements.in 
#
# This file is autogenerated by pip-compile
# Make changes in requirements.in, then run this to update:
#
#    pip-compile requirements.in
#
requests==2.9.1
alex@spicylegato:~$ mkvirtualenv test
New python executable in /home/alex/.virtualenvs/test/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/alex/.virtualenvs/test/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/alex/.virtualenvs/test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/alex/.virtualenvs/test/bin/preactivate
virtualenvwrapper.user_scripts creating /home/alex/.virtualenvs/test/bin/postactivate
virtualenvwrapper.user_scripts creating /home/alex/.virtualenvs/test/bin/get_env_details
(test) alex@spicylegato:~$ pip list
pip (8.0.3)
setuptools (20.2.2)
wheel (0.29.0)
(test) alex@spicylegato:~$ pip-sync -n requirements.txt 
Would uninstall:
  cryptography
  pymysql
  virtualenvwrapper
  pygments
  httpie
  virtualenv
  argparse
  pbr
  virtualenv-clone
  wsgiref
  stevedore
  pyasn1

This might not be the case for other environments, but the first line of my pip-sync entry point script (/usr/local/bin/pip-sync) is:

#!/usr/bin/python

This forces sync.py to import the user-installed pip, causing it to only recognize user-installed distributions.

I see 2 solutions here:

1. Use env in the shebang of the entry point script, forcing it to locate the currently active python executable:

python #!/usr/bin/env python

This seems like the cleanest solution to me. I have no idea how to do this, though, since the script is auto-generated.

1. If we can determine that a virtualenv is active, update the system path to include local site packages _before_ importing pip in sync.py. I wrote some code that does this for me, but I'm not sure if it would work for everyone.

[allanlewis]

I'm seeing the same behaviour on Fedora 23:

$ python --version
Python 2.7.11
$ pip --version
pip 8.1.1 from /home/foo/.virtualenvs/bar/lib/python2.7/site-packages (python 2.7)
$ pip-sync --version
pip-sync, version 1.6.4
$ which pip-sync
~/.virtualenvs/bar/bin/pip-sync
$ pip-sync -n requirements/*.txt
Would uninstall:
<list of everything in my global site>
Would install:
<list of entries in 'requirements/*.txt'>

That's in a new virtualenv created using mkvirtualenv with just pip-tools installed.

[wedi]

I can confirm this issue, too. pip-compile _1.6.5_ kept complaining about the wrong pip version after downgrading my virtual environment pip to _8.1.1_

[wedi]

Just for information: the issue persists on Mac OS X in 1.7.0.

[Groxx]

fwiw this is true for _every_ python binary. e.g. having fabric installed globally means you may run fabfile.py files against the global environment instead of whatever virtualenv you're in.

It might still make sense to detect a virtualenv and do something smarter! I'm just not aware of any that do so.

[Triquetra]

I am also experiencing this issue on Windows 10 with Python 3.6 in a (python -m venv) virtualenv. pip-compile seems to work fine, but pip-sync wants to uninstall all my global site-packages.

[KeynesYouDigIt]

Im having the same problem on ubuntu 16.04 ...

Cannot uninstall requirement adium-theme-ubuntu, not installed

[vphilippon]

(Catching up on stuff)
Alright, so I assume everyone here has pip-tools installed in the root python environment, and not in a the project virtualenv?

If so, that's a recipe for disaster. There's a PR coming in to update the doc and state that (better late than never...), but you should install and run pip-tools' commands in and from your project virtualenv. Cross-virtualenv stuff is a mess.

I hope that clarifies and helps!