pipenv install fails on windows with a -e git+http repository path
Describe you environment
- OS Type: Windows 10 x64
- Python version: Python 3.6.3 (32 bit)
- Pipenv version: pipenv, version 8.2.7
- git version: git version 2.14.2.windows.2
Expected result
No exception.
Actual result
> pipenv install --verbose
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Exception ignored in: <finalize object at 0x4637228; dead>
Traceback (most recent call last):
File "d:\python36-32\lib\weakref.py", line 548, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "d:\python36-32\lib\tempfile.py", line 797, in _cleanup
_shutil.rmtree(name)
File "d:\python36-32\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
[Previous line repeated 2 more times]
File "d:\python36-32\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "d:\python36-32\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dildog\\AppData\\Local\\Temp\\tmpebfeqz54source\\hidapi\\.gi
t\\modules\\hidapi\\objects\\pack\\pack-0a555dd332b5157d6b4c17ce683903db1e0ce8bd.idx'
Updated Pipfile.lock (0245b6)!
Installing dependencies from Pipfile.lock (0245b6)…
Installing 'click==6.7 --hash=sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d --hash=sha256:f15
516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b'
$ "C:\Users\dildog\.virtualenvs\bcmtool-WeLtC2nH\Scripts\pip.exe" install --verbose --no-deps -r C:\Users\dildog\AppD
ata\Local\Temp\pipenv-xq9xnj3q-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'colorama==0.3.9 --hash=sha256:463f8483208e921368c9f306094eb6f725c6ca42b0f97e313cb5d5512459feda --hash=sha25
6:48eb22f4f8461b1df5734a074b57042430fb06e1d61bd1e11b078c0fe6d7a1f1'
$ "C:\Users\dildog\.virtualenvs\bcmtool-WeLtC2nH\Scripts\pip.exe" install --verbose --no-deps -r C:\Users\dildog\AppD
ata\Local\Temp\pipenv-4q5i8m0e-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Requirement already satisfied: click==6.7 in c:\users\dildog\.virtualenvs\bcmtool-weltc2nh\lib\site-packages (from -r C:
\Users\dildog\AppData\Local\Temp\pipenv-xq9xnj3q-requirement.txt (line 1))
Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatab
le environment, install into an empty virtualenv.
Cleaning up...
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Requirement already satisfied: colorama==0.3.9 in c:\users\dildog\.virtualenvs\bcmtool-weltc2nh\lib\site-packages (from
-r C:\Users\dildog\AppData\Local\Temp\pipenv-4q5i8m0e-requirement.txt (line 1))
Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatab
le environment, install into an empty virtualenv.
Cleaning up...
================================ 2/2 - 00:00:00
Steps to replicate
Create an empty directory, add Pipfile with contents:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
click = "*"
colorama = "*"
"-e git+http://github.com/crioux/cython-hidapi#egg=hidapi" = "*"
[dev-packages]
[requires]
python_version = "3.6"
and from that folder, just run 'pipenv install'
It takes a really long time (probably building from git?) and then dies with the exception, having not fully installed the git repo dependency.
crioux
All 11 comments
Can reproduce, that's an odd one for sure :) if nobody else looks i will have a look in the morning
techalchemy
on 14 Oct 2017
Ah right, I mean to ask you -- can you show me the output of pip install -e instead of using pipenv? I don't think this is a pipenv issue
techalchemy
on 15 Oct 2017
@techalchemy any update to this? Gonna mark is as a bug now can reproduce it as well! Is this windows specific?
erinxocon
on 18 Oct 2017
@erinxocon I tried to install with pipenv and the error was not super detailed, but when I tried with pip the error was more along the lines of requiring a VC++ compiler of a specific variety. If github had a reminder functionality I would remind myself to post the output here tomorrow but the issue here may just be that we don't pass the error message through properly
techalchemy
on 18 Oct 2017
I'm going to close this as we never heard back and it was related to VC++ compiler libraries on windows
techalchemy
on 8 Nov 2017
I'm getting this error message and it doesn't appear to be due to VC++ compiler libraries. My pipfile has dnc = {git = "git://github.com/edbordin/dnc.git", editable = true, ref = "master"} and every time I run pipenv install to add a new package, I get the following:
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Exception ignored in: <finalize object at 0x1c272186050; dead>
Traceback (most recent call last):
File "c:\python36-x64\lib\weakref.py", line 548, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "c:\python36-x64\lib\tempfile.py", line 797, in _cleanup
_shutil.rmtree(name)
File "c:\python36-x64\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\python36-x64\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\python36-x64\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\python36-x64\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\python36-x64\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "c:\python36-x64\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\edwardb\\AppData\\Local\\Temp\\tmpb9u8h4xisource\\dnc\\.git\\objects\\pack\\pack-f17a396344daec8ce6fbb520cd8f2edc6b128593.idx'
Updated Pipfile.lock (864952)!
If I create a new virtualenv with Python 3.6 and run pip manually it installs OK:
D:\dev>mkvirtualenv test
Using base prefix 'c:\\python36-x64'
New python executable in D:\dev\python-env\test\Scripts\python.exe
Installing setuptools, pip, wheel...done.
(test) D:\dev>pip install -e git://github.com/edbordin/dnc.git@8763cce0cf993f927db791b1374c96a8997225b0#egg=dnc
Obtaining dnc from git+git://github.com/edbordin/dnc.git@8763cce0cf993f927db791b1374c96a8997225b0#egg=dnc
Cloning git://github.com/edbordin/dnc.git (to 8763cce0cf993f927db791b1374c96a8997225b0) to d:\dev\python-env\testegg\src\dnc
Could not find a tag or branch '8763cce0cf993f927db791b1374c96a8997225b0', assuming commit.
Collecting dm-sonnet (from dnc)
Using cached dm_sonnet-1.16-py3-none-any.whl
Collecting six (from dm-sonnet->dnc)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting absl-py (from dm-sonnet->dnc)
Installing collected packages: six, absl-py, dm-sonnet, dnc
Running setup.py develop for dnc
Successfully installed absl-py-0.1.10 dm-sonnet-1.16 dnc six-1.11.0
ed-alertedh
on 14 Feb 2018
@ed-alertedh that is not a vc++ compiler issue but as it says, it is a permissions issue, which is why a fresh clone works. Git indexes clone in read only mode so you can't unlink them (this is a massive nightmare jon windows, speaking from experience). This isn't a pipenv issue but rather an issue with windows file permissions in general, and with git index permissions specifically. You can manually change the permissions on that path and it should solve your problem.
techalchemy
on 14 Feb 2018
Right, I totally agree that NTFS permissions are a basket case. But requiring the user to manually change the permissions is really not ideal - this appears to be a workaround for Windows https://stackoverflow.com/a/4829285/7706514
I'm having trouble telling if the rmtree is getting called by pipenv or by pip. If it's the former I'd be happy to attempt putting together a PR with the above workaround.
ed-alertedh
on 14 Feb 2018
It isn't called directly by either actually, as your exception indicates it is called indirectly by tempfile which is supposed to remove this temporary resource which was downloaded (I happen to know that it was downloaded by pip because I have already spent several ~hours~ days working on this specific issue) as part of the dependency resolution process.
I realize that at first glance it may seem that the SO answer provided would be sufficient to fix this issue, but actually that only scratches the surface. In particular, this issue is related to the way subprocess.Popen works on windows. When you call it with shell=True (as we do), it spawns a new process _group_, which in this case invokes pip to do some stuff. In the case of editable VCS dependencies, pip then does this again, invoking _git_ to grab the repository. On Windows in particular, Popen leaks open file descriptors, which means that the spawned process group of the spawned process group has a leaked open file descriptor to the git index you're trying to delete.
This doesn't happen ALL the time, but only sometimes. And when it does, it means that there is no universe in which you can do anything with native error handling to fix this, because the FD is still open. Now you might be thinking, sure, I'll use psutil, find all the things with open file descriptors on that index, kill those process groups, and then set the permissions, then I can delete them. I have been down that road.
This is all somewhat irrelevant because I have a fix sitting in a PR which is held up currently due to #1372 -- I believe this might be what you are interested in
techalchemy
on 14 Feb 2018
Well in that case, excellent debugging work and I'm happy to hear there is a fix waiting to be merged! The context manager you've implemented looks like a nice solution within the constraints you described - falling back to a warning when an open fd has leaked is definitely much nicer than the current traceback.
Apologies for assuming it was a simple fix, I didn't realise somebody was working on it as the issue was closed and not referenced anywhere else. I was mainly trying to sound less whiny by offering to help :)
ed-alertedh
on 14 Feb 2018
I assumed it was a simple fix too before I wound up down that rabbit hole, what a nightmare that whole process was
techalchemy
on 14 Feb 2018
Related issues
ansrivas
·
3Comments
fbender
·
3Comments
bgjelstrup
·
3Comments
randName
·
3Comments
marc-fez
·
3Comments
Most helpful comment
It isn't called directly by either actually, as your exception indicates it is called indirectly by
tempfilewhich is supposed to remove this temporary resource which was downloaded (I happen to know that it was downloaded by pip because I have already spent several ~hours~ days working on this specific issue) as part of the dependency resolution process.I realize that at first glance it may seem that the SO answer provided would be sufficient to fix this issue, but actually that only scratches the surface. In particular, this issue is related to the way
subprocess.Popenworks on windows. When you call it withshell=True(as we do), it spawns a new process _group_, which in this case invokes pip to do some stuff. In the case of editable VCS dependencies, pip then does this again, invoking _git_ to grab the repository. On Windows in particular,Popenleaks open file descriptors, which means that the spawned process group of the spawned process group has a leaked open file descriptor to the git index you're trying to delete.This doesn't happen ALL the time, but only sometimes. And when it does, it means that there is no universe in which you can do anything with native error handling to fix this, because the FD is still open. Now you might be thinking, sure, I'll use psutil, find all the things with open file descriptors on that index, kill those process groups, and then set the permissions, then I can delete them. I have been down that road.
This is all somewhat irrelevant because I have a fix sitting in a PR which is held up currently due to #1372 -- I believe this might be what you are interested in