pip uninstall not working with editable dependencies
- Pip version: 10.0.0b2 as well as 10.0.0b2+ (8e074408d69f4b1e064dbc8908f1ec7dcf034be5)
- Python version: 2.7.12
- Operating system: Ubuntu 16.04
Description:
After installing an editable dependency from vcs, it cannot be uninstalled.
With pip 9 it did uninstall correctly, removing the .egg-link file.
What I've run:
$ virtualenv --version
15.0.1
$ virtualenv venv
$ venv/bin/pip --version
10.0.0b2 # wondering why I'm getting a beta version of pip without asking but that's not the point here
$ venv/bin/pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Updating ./venv/src/click clone
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
$ venv/bin/pip uninstall click
Can't uninstall 'click'. No files were found to uninstall.
$ venv/bin/pip list
Package Version Location
------------- -------- ----------------------------------
click 7.0.dev0 /home/sbi-local/tmp/venv/src/click
pip 10.0.0b2
pkg-resources 0.0.0
setuptools 39.0.1
wheel 0.31.0
I expected click to be uninstalled at this point.
If I downgrade to pip 9.0.3, it works:
$ venv/bin/pip install pip==9.0.3
Collecting pip==9.0.3
Using cached pip-9.0.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 10.0.0b2
Uninstalling pip-10.0.0b2:
Successfully uninstalled pip-10.0.0b2
Successfully installed pip-9.0.3
$ venv/bin/pip uninstall click
Uninstalling click-7.0.dev0:
/home/sbi-local/tmp/venv/lib/python2.7/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
$ venv/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.3)
pkg-resources (0.0.0)
setuptools (39.0.1)
wheel (0.31.0)
sbidoul
All 45 comments
I can't reproduce this with virtualenv==15.0.1, virtualenv==15.2.0, or pyenv:
$ pip install virtualenv==15.0.1
Collecting virtualenv==15.0.1
Using cached virtualenv-15.0.1-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-15.0.1
$ python -m virtualenv env
Using base prefix '/Users/di/.pyenv/versions/3.6.4'
New python executable in /private/tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ source env/bin/activate
(env) $ pip install -U pip --pre
Requirement already up-to-date: pip in ./env/lib/python3.6/site-packages (10.0.0b2)
(env) $ pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Cloning https://github.com/pallets/click.git to ./env/src/click
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
(env) $ pip uninstall click
Uninstalling click-7.0.dev0:
Would remove:
/private/tmp/env/lib/python3.6/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
(env) $ pip freeze
$ pip install virtualenv
Collecting virtualenv
Using cached virtualenv-15.2.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-15.2.0
$ python -m virtualenv env
Using base prefix '/Users/di/.pyenv/versions/3.6.4'
New python executable in /private/tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ source env/bin/activate
(env) $ pip install -U pip --pre
Collecting pip
Using cached pip-10.0.0b2-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.0b2
(env) $ pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Cloning https://github.com/pallets/click.git to ./env/src/click
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
(env) $ pip uninstall click
Uninstalling click-7.0.dev0:
Would remove:
/private/tmp/env/lib/python3.6/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
(env) $ pip freeze
$ python -m venv env
$ source env/bin/activate (env) $ pip install -U pip --pre
Collecting pip
Using cached pip-10.0.0b2-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.0b2
(env) $ pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Cloning https://github.com/pallets/click.git to ./env/src/click
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
(env) $ pip uninstall click
Uninstalling click-7.0.dev0:
Would remove:
/private/tmp/env/lib/python3.6/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
(env) $ pip freeze
di
on 13 Apr 2018
It looks like this issue is python 2 specific.
sbidoul
on 13 Apr 2018
Still can't reproduce:
$ python2 -m virtualenv env
New python executable in /private/tmp/env/bin/python2
Also creating executable in /private/tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ source env/bin/activate
(env) /tmp $ virtualenv --version
15.0.1
(env) /tmp $ python --version
Python 2.7.12
(env) $ pip --version
pip 10.0.0b2 from /private/tmp/env/lib/python2.7/site-packages/pip (python 2.7)
(env) $ pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Cloning https://github.com/pallets/click.git to ./env/src/click
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
(env) $ pip uninstall click
Uninstalling click-7.0.dev0:
Would remove:
/private/tmp/env/lib/python2.7/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
(env) $ pip freeze
$ python2 -m virtualenv env
New python executable in /private/tmp/env/bin/python2
Also creating executable in /private/tmp/env/bin/python
Installing setuptools, pip, wheel...done.
$ source env/bin/activate
(env) $ virtualenv --version
15.2.0
(env) $ python --version
Python 2.7.12
(env) $ pip --version
pip 10.0.0b2 from /private/tmp/env/lib/python2.7/site-packages/pip (python 2.7)
(env) $ pip install -e git+https://github.com/pallets/click.git#egg=click
Obtaining click from git+https://github.com/pallets/click.git#egg=click
Cloning https://github.com/pallets/click.git to ./env/src/click
Installing collected packages: click
Running setup.py develop for click
Successfully installed click
(env) $ pip uninstall click
Uninstalling click-7.0.dev0:
Would remove:
/private/tmp/env/lib/python2.7/site-packages/click.egg-link
Proceed (y/n)? y
Successfully uninstalled click-7.0.dev0
(env) $ pip freeze
Also, still with pyhon 2.7, when I pip install with --src, I get the following error:
$ pip install -e git+https://github.com/pallets/click.git#egg=click --src src
$ pip uninstall click
Not uninstalling click at /home/sbi-local/tmp/src/click, outside environment /home/sbi-local/tmp/venv2
Can't uninstall 'click'. No files were found to uninstall.
No such issue with python 3.
sbidoul
on 13 Apr 2018
@di that is weird, I can reproduce consistently with python 2.
I just reproduced my instructions above in a fresh ubuntu 16.04 lxc container:
$ lxc launch ubuntu:16.04 ub
$ lxc exec ub bash
# apt update && apt install python virtualenv
# virtualenv venv
# source venv/bin/activate
# pip install -e git+https://github.com/pallets/click.git#egg=click
# pip uninstall click
Can't uninstall 'click'. No files were found to uninstall.
# virtualenv --version
15.0.1
# python --version
Python 2.7.12
# pip --version
pip 10.0.0b2 from /root/venv/local/lib/python2.7/site-packages/pip (python 2.7)
I've also encountered the problem, however in my case the error message is slightly different:
$ pip uninstall pytest-catchlog
Not uninstalling pytest-catchlog at /home/mg/src/pytest-catchlog, outside environment /home/mg/.venv
Can't uninstall 'pytest-catchlog'. No files were found to uninstall.
Steps to reproduce:
$ virtualenv --system-site-packages ~/.venv
$ PATH=~/.venv/bin:$PATH
$ pip install -U --pre pip
$ pip --version
pip 10.0.0b2 from /home/mg/.venv/local/lib/python2.7/site-packages/pip (python 2.7)
$ pip install -e ~/src/pytest-catchlog
Obtaining file:///home/mg/src/pytest-catchlog
...
Installing collected packages: pytest-catchlog
Running setup.py develop for pytest-catchlog
Successfully installed pytest-catchlog
$ pip uninstall pytest-catchlog
Not uninstalling pytest-catchlog at /home/mg/src/pytest-catchlog, outside environment /home/mg/.venv
Can't uninstall 'pytest-catchlog'. No files were found to uninstall.
$ pip list|grep pytest-catchlog
pytest-catchlog 1.2.2 /home/mg/src/pytest-catchlog
If I downgrade pip to version 9.0.3, I can pip uninstall editable packages in the virtualenv
$ pip install 'pip<10'
...
Successfully installed pip-9.0.3
You are using pip version 9.0.3, however version 10.0.0b2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ $ pip uninstall pytest-catchlog
Uninstalling pytest-catchlog-1.2.2:
/home/mg/.venv/lib/python2.7/site-packages/pytest-catchlog.egg-link
Proceed (y/n)? y
Successfully uninstalled pytest-catchlog-1.2.2
(~/src/pytest-catchlog is a checkout of https://github.com/eisensheng/pytest-catchlog)
mgedmin
on 13 Apr 2018
@mgedmin What platform is this? What version of virtualenv did you use?
di
on 13 Apr 2018
I'm on Ubuntu 17.10. I cannot say for sure which virtualenv version I used: ~/.venv/ was created a long time ago (Sep 10 2016, if I can trust the timestamp of the 'python2.7' symlink in ~/.venv/bin).
mgedmin
on 13 Apr 2018
I can reproduce using the latest virtualenv:
mg@platonas: ~ $ pip install -U virtualenv
Requirement already up-to-date: virtualenv in ./.venv/lib/python2.7/site-packages (15.2.0)
mg@platonas: ~ $ rm -rf /tmp/sandbox/; virtualenv -p /usr/bin/python2.7 /tmp/sandbox
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /tmp/sandbox/bin/python2.7
Also creating executable in /tmp/sandbox/bin/python
Installing setuptools, pip, wheel...done.
mg@platonas: ~ $ /tmp/sandbox/bin/pip install -U pip
Requirement already up-to-date: pip in /tmp/sandbox/lib/python2.7/site-packages
mg@platonas: ~ $ /tmp/sandbox/bin/pip --version
pip 9.0.3 from /tmp/sandbox/local/lib/python2.7/site-packages (python 2.7)
mg@platonas: ~ $ /tmp/sandbox/bin/pip install -U pip --pre
Collecting pip
Using cached pip-10.0.0b2-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.0b2
mg@platonas: ~ $ /tmp/sandbox/bin/pip install -e ~/src/pytest-catchlog/
Obtaining file:///home/mg/src/pytest-catchlog
Collecting py>=1.1.1 (from pytest-catchlog==1.2.2)
Using cached py-1.5.3-py2.py3-none-any.whl
Collecting pytest>=2.6 (from pytest-catchlog==1.2.2)
Using cached pytest-3.5.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /tmp/sandbox/lib/python2.7/site-packages (from pytest>=2.6->pytest-catchlog==1.2.2) (39.0.1)
Collecting more-itertools>=4.0.0 (from pytest>=2.6->pytest-catchlog==1.2.2)
Using cached more_itertools-4.1.0-py2-none-any.whl
Collecting funcsigs; python_version < "3.0" (from pytest>=2.6->pytest-catchlog==1.2.2)
Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting six>=1.10.0 (from pytest>=2.6->pytest-catchlog==1.2.2)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from pytest>=2.6->pytest-catchlog==1.2.2)
Using cached attrs-17.4.0-py2.py3-none-any.whl
Collecting pluggy<0.7,>=0.5 (from pytest>=2.6->pytest-catchlog==1.2.2)
Installing collected packages: py, six, more-itertools, funcsigs, attrs, pluggy, pytest, pytest-catchlog
The scripts py.test and pytest are installed in '/tmp/sandbox/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py develop for pytest-catchlog
Successfully installed attrs-17.4.0 funcsigs-1.0.2 more-itertools-4.1.0 pluggy-0.6.0 py-1.5.3 pytest-3.5.0 pytest-catchlog six-1.11.0
mg@platonas: ~ $ /tmp/sandbox/bin/pip uninstall pytest-catchlog
Not uninstalling pytest-catchlog at /home/mg/src/pytest-catchlog, outside environment /tmp/sandbox
Can't uninstall 'pytest-catchlog'. No files were found to uninstall.
I could reproduce this in a fresh clean Docker container: https://gist.github.com/mgedmin/0d5f8cadac30270a997c3a7b3bd69bf6
mgedmin
on 13 Apr 2018
It looks like this issue is python 2 specific.
I too was unable to reproduce this using Python 3.
mgedmin
on 13 Apr 2018
I can reproduce this issue under Ubuntu 16.04 with Python 2.7.12 and virtualenv 15.2.0. After downgrading to pip 9.0.3 I can once again uninstall editable packages.
ryan-rs
on 19 Apr 2018
I've created a PR for pip-tools to show this issue on Python 2.7 / Ubuntu 16. Run tox -e py27 from that PR on Ubuntu to reproduce.
tysonclugg
on 2 May 2018
I am unable to get the first bug to show up however the one mentioned by @sbidoul consistently breaks on python2.
@di you are failing to reproduce because when you do the pip install -e git+https://... it clones the repo in the virtualenv folder under src, try using the --src flag.
bucknerns
on 3 May 2018
Well, consider myself nerdsniped 😆
I attempted to bisect this, hope this is helpful :)
#!/usr/bin/env bash
set -euxo pipefail
export PIP_DISABLE_PIP_VERSION_CHECK=1 VIRTUALENV_NO_DOWNLOAD=1
finish() {
git checkout -- .
git clean -fxfd
}
trap finish ERR EXIT
# hotfix for `AttributeError: 'module' object has no attribute 'get_python_lib'
# cherry-picked from 310bcfc78fea8f1f2bc4f680b376e18bf5aeed07
if \
grep -q '^import sysconfig$' pip/locations.py && \
grep -q 'site_packages = sysconfig.get_python_lib()$' pip/locations.py; then
sed -i 's/sysconfig.get_python_lib()/sysconfig.get_path("purelib")/g' pip/locations.py
fi
mkdir tdir
(
cd tdir
echo 'from setuptools import setup; setup(name="pkg")' > setup.py
virtualenv venv -ppython2 > /dev/null
venv/bin/pip install ../ > /dev/null || exit 125
venv/bin/pip install -e . > /dev/null || exit 125
! (venv/bin/pip uninstall -y pkg | grep 'Not uninstalling pkg')
)
I had to apply the patch listed in 310bcfc78fea8f1f2bc4f680b376e18bf5aeed07 early so that the bisect was able to differentiate between a bunch of "broken" commits which ended with:
+ venv/bin/pip install -e .
Traceback (most recent call last):
File "venv/bin/pip", line 7, in <module>
from pip import main
File "/tmp/test/pip/tdir/venv/local/lib/python2.7/site-packages/pip/__init__.py", line 26, in <module>
from pip.utils import get_installed_distributions, get_prog
File "/tmp/test/pip/tdir/venv/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 23, in <module>
from pip.locations import (
File "/tmp/test/pip/tdir/venv/local/lib/python2.7/site-packages/pip/locations.py", line 83, in <module>
site_packages = sysconfig.get_python_lib()
AttributeError: 'module' object has no attribute 'get_python_lib'
git bisect start
git bisect good 9.0.3
git bisect bad HEAD
git bisect run ../bisect.sh
After the bisect run, it points at this:
904fcf1f177770a4eb05e0850d79265e4bc0907f is the first bad commit
commit 904fcf1f177770a4eb05e0850d79265e4bc0907f
Author: Donald Stufft <[email protected]>
Date: Sat Mar 18 13:32:10 2017 -0400
Use sysconfig instead of distutils.sysconfig
:040000 040000 dfd417efac444008f5ca7e07f85de83168e0e6de 710942f93e7ce4df8553c8534b515a60aa2feb39 M pip
bisect run success
CC @dstufft 904fcf1f177770a4eb05e0850d79265e4bc0907f
A further triage, the failure appears to come from a differing return value in dist_is_local here:
The difference appears to be the return values here:
$ venv/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))'
/tmp/test/venv/local/lib/python2.7/dist-packages
$ venv3/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))'
/tmp/test/venv3/lib/python3.5/site-packages
asottile
on 13 May 2018
Is there any solution for this problem?
(test_env) $ cd [HOME]/test_package
(test_env) $ pip install -e .
Installing collected packages: test_package
Running setup.py develop for test_package
Successfully installed test_package
(test_env) $ pip uninstall -y test_package
Not uninstalling test_package at [HOME]/test_package, outside environment [HOME]/env/test_env
Can't uninstall 'test_package'. No files were found to uninstall.
(test_dev) $ pip --version
pip 18.0 from [HOME]/env/test_dev/local/lib/python2.7/site-packages/pip (python 2.7)
marc1n
on 9 Aug 2018
@asottile So if sysconfig.get_path() is the culprit, might this mean there's a bug in sysconfig.get_path()'s implementation / return value? Can you provide steps to create a virtualenv where sysconfig.get_path() differs from distutils.sysconfig's -- is that the source of the problem? The example you gave at the end of your last comment shows the return values only for sysconfig.get_path() -- and for different virtualenvs (venv and venv3) rather than the same one -- so it's not clear to me what the difference in return values in that comment is meant to show..
cjerdonek
on 10 Aug 2018
@cjerdonek the key difference in the two outputs is in python2 the debianinzed dist-packages is leaking through (it's not being a real directory in virtualenvs, dist-packages is only a thing at the system level).
asottile
on 10 Aug 2018
Okay, so in other words, the following line works / gives a different answer if you substitute the distutils version?
$ venv/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))'
Here's the eight values:
FROM debian:stretch
RUN : \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
dumb-init \
python \
python3 \
virtualenv \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists
RUN : \
&& virtualenv /venv2 -ppython2.7 \
&& virtualenv /venv3 -ppython3
RUN /bin/echo -e '#!/usr/bin/env bash \n\
set -euxo pipefail \n\
/usr/bin/python -c "from distutils import sysconfig; print(sysconfig.get_python_lib())" \n\
/usr/bin/python -c "import sysconfig; print(sysconfig.get_path('"'"'purelib'"'"'))" \n\
/usr/bin/python3 -c "from distutils import sysconfig; print(sysconfig.get_python_lib())" \n\
/usr/bin/python3 -c "import sysconfig; print(sysconfig.get_path('"'"'purelib'"'"'))" \n\
/venv2/bin/python -c "from distutils import sysconfig; print(sysconfig.get_python_lib())" \n\
/venv2/bin/python -c "import sysconfig; print(sysconfig.get_path('"'"'purelib'"'"'))" \n\
/venv3/bin/python -c "from distutils import sysconfig; print(sysconfig.get_python_lib())" \n\
/venv3/bin/python -c "import sysconfig; print(sysconfig.get_path('"'"'purelib'"'"'))" \n\
' > t.sh
CMD ["dumb-init", "bash", "t.sh"]
$ docker build -q -t test . && docker run --rm -ti test
sha256:b10eb0cb39a9a9e9c641a7496e1447047b850852f1573038d3fc43280a5961ef
+ /usr/bin/python -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())'
/usr/lib/python2.7/dist-packages
+ /usr/bin/python -c 'import sysconfig; print(sysconfig.get_path('\''purelib'\''))'
/usr/local/lib/python2.7/dist-packages
+ /usr/bin/python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())'
/usr/lib/python3/dist-packages
+ /usr/bin/python3 -c 'import sysconfig; print(sysconfig.get_path('\''purelib'\''))'
/usr/lib/python3.5/site-packages
+ /venv2/bin/python -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())'
/venv2/lib/python2.7/site-packages
+ /venv2/bin/python -c 'import sysconfig; print(sysconfig.get_path('\''purelib'\''))'
/venv2/local/lib/python2.7/dist-packages
+ /venv3/bin/python -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())'
/venv3/lib/python3.5/site-packages
+ /venv3/bin/python -c 'import sysconfig; print(sysconfig.get_path('\''purelib'\''))'
/venv3/lib/python3.5/site-packages
Thanks. It looks like not just one but three of the four pairs have different return values (and for different reasons), with only the last the same. So does this mean that one (or perhaps both) of the two functions sysconfig.get_path() and distutils.sysconfig.get_python_lib() has a bug, or are they not supposed to behave the same?
cjerdonek
on 10 Aug 2018
Debian system python installs a symlink to make this work there
On Fri, Aug 10, 2018, 3:56 AM Chris Jerdonek notifications@github.com
wrote:
Thanks. It looks like not just one but three of the four pairs have
different return values (and for different reasons), with only the last the
same. So does this mean that one (or perhaps both) of the two functions
sysconfig.get_path() and distutils.sysconfig.get_python_lib() has a bug,
or are they not supposed to behave the same?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/5193#issuecomment-412049417, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABugn7x9otsHDQrC3p8ThqiFW1wWqKKOks5uPWbHgaJpZM4TLm1R
.
asottile
on 10 Aug 2018
My question was whether the two functions are always supposed to have the same return value.
cjerdonek
on 10 Aug 2018
unclear to me, but one was swapped for the other in https://github.com/pypa/pip/commit/904fcf1f177770a4eb05e0850d79265e4bc0907f so that was at least assumed by @dstufft
asottile
on 10 Aug 2018
I would imagine that they are intended to give the same result (I think sysconfig was added to the stdlib to pull certain generally-useful bits out of distutils into a more generic location). But Debian's messing round with system paths always ends up causing us problems. In this case, it may be that the amount of Debian specific special-casing already in virtualenv isn't sufficient to catch this case.
pfmoore
on 10 Aug 2018
So is this issue an upstream issue with virtualenv? Has an issue already been filed? This previous virtualenv issue seems related as it's also about sysconfig under Python 2.7 and references Debian special-casing: https://github.com/pypa/virtualenv/issues/118
cjerdonek
on 12 Aug 2018
I'm not 100% sure, but I'd say that pip is OK to expect sysconfig.get_path() and distutils.sysconfig.get_python_lib() to behave the same, so if they aren't, that's a problem upstream. The problem seems to be related to Debian's modification of standard Python file locations, which means that my immediate instinct is to say it's Debian upstream that's the issue here, but virtualenv does have code in already to work with Debian's changed paths, so it may be that virtualenv's code is the place where a change is needed. Virtualenv is basically a huge collection of hacks to work around distribution idiosyncracies, so it's quite likely that it is the place where something like this would get addressed.
But I'm not an expert on Debian, much less on how people handle their customisations, so my comments are basically just an educated guess here.
pfmoore
on 12 Aug 2018
I was trying to find debian issue regarding this problem but with no luck. Focusing on current stable debian (stretch) all debian patches applied to Python (version 3.5.3-1 as of now) are listed at https://sources.debian.org/patches/python3.5/3.5.3-1/ There, we can see there's https://sources.debian.org/patches/python3.5/3.5.3-1/distutils-install-layout.diff set of patches which includes patches both for Lib/distutils/sysconfig.py and for Lib/site.py but does not have a patch for Lib/sysconfig.py. This could explain the difference in behavior between distutils.sysconfig.get_python_lib() and sysconfig.get_path('purelib').
For reference, get_python_lib() function from distutils.sysconfig with debian's patch applied (https://sources.debian.org/src/python3.5/3.5.3-1/Lib/distutils/sysconfig.py/#L118-L160):
def get_python_lib(plat_specific=0, standard_lib=0, prefix=None):
"""Return the directory containing the Python library (standard or
site additions).
If 'plat_specific' is true, return the directory containing
platform-specific modules, i.e. any module from a non-pure-Python
module distribution; otherwise, return the platform-shared library
directory. If 'standard_lib' is true, return the directory
containing standard Python library modules; otherwise, return the
directory for site-specific modules.
If 'prefix' is supplied, use it instead of sys.base_prefix or
sys.base_exec_prefix -- i.e., ignore 'plat_specific'.
"""
is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
if prefix is None:
if standard_lib:
prefix = plat_specific and BASE_EXEC_PREFIX or BASE_PREFIX
else:
prefix = plat_specific and EXEC_PREFIX or PREFIX
if os.name == "posix":
libpython = os.path.join(prefix,
"lib", "python" + get_python_version())
if standard_lib:
return libpython
elif (is_default_prefix and
'PYTHONUSERBASE' not in os.environ and
'VIRTUAL_ENV' not in os.environ and
'real_prefix' not in sys.__dict__ and
sys.prefix == sys.base_prefix):
return os.path.join(prefix, "lib", "python3", "dist-packages")
else:
return os.path.join(libpython, "site-packages")
elif os.name == "nt":
if standard_lib:
return os.path.join(prefix, "Lib")
else:
return os.path.join(prefix, "Lib", "site-packages")
else:
raise DistutilsPlatformError(
"I don't know where Python installs its library "
"on platform '%s'" % os.name)
piotr-dobrogost
on 21 Aug 2018
I propose a workaround in #6918 (a similar workaround is already in place for pypy).
sbidoul
on 25 Aug 2019
@kitterma This seems to be an issue introduced by Debian's patch for dist-packages. It would be wonderful if you could look into introducing a fix in Debian. :)
pradyunsg
on 23 May 2020
I took a look a this on Debian stable as well as unstable/testing. Click is now python3 only so the original problem with that repository can't occur. I tried the reproduction steps on python3 and it uninstalled fine.
I don't know how much effort it's worth fixing python2 stuff now, but if someone will provide another example where the problem still occurs I will take a look at it.
kitterma
on 23 May 2020
@pradyunsg this was caused by a change in pip fwiw: https://github.com/pypa/pip/issues/5193#issuecomment-388651853
asottile
on 23 May 2020
I think the main issue is why a platform incorrectly returns dist-packages in sysconfig though. Is it patched? Why is it patched? Should the patch be fixed instead?
uranusjr
on 23 May 2020
🤷 it used to work though is my point
asottile
on 23 May 2020
As I said earlier, if someone can give me a current example showing the problem, I'll look into it.
kitterma
on 23 May 2020
@kitterma this should reproduce on buster.
$ mkdir editabletest
$ cd editabletest
$ echo "import setuptools; setuptools.setup(name='editabletest')" > setup.py
$ virtualenv -p python2 venv
$ venv/bin/pip install -e .
$ venv/bin/pip uninstall editabletest
...
Not uninstalling editabletest at /root/editabletest, outside environment /root/editabletest/venv
...
I thought about this a bit more. Python has two ways to get information on where to install stuff: distutils.sysconfig (legacy), and sysconfig (modern). The two usually should match, but may not be if one of them is incorrectly patched in an environment.
pip should be enforcing the modern scheme when it installs stuff, but can likely be more lenient when uninstalling. It could be possible for the uninstallation logic to try figure out which scheme a package is installed with, and use the right one automatically. This is worthwhilte to investigate IMO since it is still quite common for packages to be installed with the legacy scheme, either by an old pip (9.x is still in active use!) or an installation method that has not been modernised (e.g. editable).
I’ll probably take a deeper look at the implementation on Sunday if nobody else figures this out before that.
uranusjr
on 23 May 2020
@kitterma this should reproduce on buster.
$ mkdir editabletest $ cd editabletest $ echo "import setuptools; setuptools.setup(name='editabletest')" > setup.py $ virtualenv -p python2 venv $ venv/bin/pip install -e . $ venv/bin/pip uninstall editabletest ... Not uninstalling editabletest at /root/editabletest, outside environment /root/editabletest/venv ...
Thanks. Same thing happens on Unstable/Testing with python2.7. With python3.8 it works fine.
kitterma
on 23 May 2020
Looking into this a bit more. On Debian Unstable, inside a python2 based virtualenv I see:
>>> import distutils.sysconfig
>>> print distutils.sysconfig.get_python_lib()
/$HOMEDIR/editabletest/venv/lib/python2.7/site-packages
>>> import sysconfig
>>> print sysconfig.get_path('purelib')
/$HOMEDIR/editabletest/venv/local/lib/python2.7/dist-packages
In a python3 virtualenv it's different:
>>> import distutils.sysconfig
>>> print(distutils.sysconfig.get_python_lib())
/$HOMEDIR/editabletest/venv3/lib/python3.8/site-packages
>>> import sysconfig
>>> print(sysconfig.get_path('purelib'))
/$HOMEDIR/editabletest/venv3/lib/python3.8/site-packages
So it's pretty clear why python3 works and python2 doesn't. Even though this may have be precipitated by a pip change, I think downstream is the correct resolution since it's connected to a Debian specific Python interpreter change (which isn't going to go away).
kitterma
on 23 May 2020
Just to confirm, has anyone in this thread tried to file an issue to Debian?
uranusjr
on 24 May 2020
On Saturday, May 23, 2020 8:06:13 PM EDT Tzu-ping Chung wrote:
Just to confirm, has anyone in this thread tried to file an issue to Debian?
Since I'm the primary maintainer for virtualenv and pip in Debian right now,
it wouldn't make me any more aware of it than I am now.
Also, don't assume the previous discussion is right. I tried a patch very
similar to the linked pull request and it didn't solve the problem. That may
be on me for either not getting the change right or not testing correctly, but
I'm not 100% sure the previous discussion is on the right path.
kitterma
on 24 May 2020
I see, thanks @kitterma.
uranusjr
on 24 May 2020
@kitterma thanks for your interest in this issue.
I rebased #6918 and I confirm it still fixes it:
console
$ mkdir editabletest
$ cd editabletest
$ echo "import setuptools; setuptools.setup(name='editabletest')" > setup.py
$ virtualenv -p python2 venv
$ venv/bin/pip install https://github.com/sbidoul/pip/archive/editable-uninstall-2.7-sbi.zip
$ venv/bin/pip install -e .
$ venv/bin/pip uninstall editabletest
Found existing installation: editabletest 0.0.0
Uninstalling editabletest-0.0.0:
Would remove:
/.../editabletest/venv/lib/python2.7/site-packages/editabletest.egg-link
Proceed (y/n)? y
Successfully uninstalled editabletest-0.0.0
sbidoul
on 24 May 2020
I can confirm it too now. I'm not sure what I was doing wrong before. Thanks.
I will include this in the next pip upload for Debian Unstable, which means it will support Debian Bullseye. Since the Buster (Debian 10) version of virtualenv always uses the latest version of pip from pypi, I don't think there's any benefit to me backporting the patch (please let me know if that's wrong).
kitterma
on 24 May 2020
I can confirm I have the bug on python2.7.
What's the status of this ? @kitterma
rousseldenis
on 4 Sep 2020
Related issues
Zac-HD
·
3Comments
kkom
·
3Comments
jiapei100
·
3Comments
nathan130200
·
3Comments
lululukas
·
3Comments
Most helpful comment
Well, consider myself nerdsniped 😆
I attempted to bisect this, hope this is helpful :)
I had to apply the patch listed in 310bcfc78fea8f1f2bc4f680b376e18bf5aeed07 early so that the bisect was able to differentiate between a bunch of "broken" commits which ended with:
After the bisect run, it points at this:
CC @dstufft 904fcf1f177770a4eb05e0850d79265e4bc0907f
A further triage, the failure appears to come from a differing return value in
dist_is_localhere:https://github.com/pypa/pip/blob/d67d98dd914e2ce80ece43594554f0a226558db0/src/pip/_internal/req/req_uninstall.py#L276
The difference appears to be the return values here: