pip install -U . fails when there's a UNIX domain socket in the current directory

[gvanrossum]

• Pip version: 10.0.1
• Python version: 3.6
• Operating system: macOS Sierra

Description:

It seems pip 10.0.1, when installing from a directory, makes a copy of that directory before doing anything else (?). This causes problems for mypy developers. We have a daemon process that creates a UNIX domain socket named dmypy.sock in the current directory. When the mypy daemon is running in the mypy directory, and we try to install from there, this copy fails with the following error:

$ pip3 install -U .
Processing /Users/guido/src/mypy
Could not install packages due to an EnvironmentError: [('/Users/guido/src/mypy/dmypy.sock', '/private/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/pip-req-build-yucol_r3/dmypy.sock', "[Errno 102] Operation not supported on socket: '/Users/guido/src/mypy/dmypy.sock'")]

Using -v we get the following traceback:

$ pip3 install -U . -v
Created temporary directory: /private/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/pip-ephem-wheel-cache-z05l_dty
Created temporary directory: /private/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/pip-install-pdgnid0m
Processing /Users/guido/src/mypy
  Created temporary directory: /private/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/pip-req-build-cykzr90d
Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 310, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/download.py", line 824, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/Users/guido/v36/lib/python3.6/site-packages/pip/_internal/download.py", line 700, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/shutil.py", line 359, in copytree
    raise Error(errors)
shutil.Error: [('/Users/guido/src/mypy/dmypy.sock', '/private/var/folders/63/czkyq6090dd0t157zhx54xvhrdlybt/T/pip-req-build-cykzr90d/dmypy.sock', "[Errno 102] Operation not supported on socket: '/Users/guido/src/mypy/dmypy.sock'")]
Cleaning up...

A simple workaround is to stop the daemon (or if it's been killed, rm dmypy.sock).

(First reported for mypy: https://github.com/python/mypy/issues/4945, for @ilevkivsky.)

[pradyunsg]

A fix for this would be that pip tries to be smarter about what it's copying.

Honestly though, I have a feeling that adding support for a .pipignore will let people work around edge cases and that would be cleaner than adding support for all kinds of edge cases in pip.

Unrelated: pip will always reinstall from a local directory -- pip install . should just work for you. :)

[gvanrossum]

It's also arguably a bug in shutil.copytree(), which contains this comment:

# XXX What about other special files? (sockets, devices...)

[gvanrossum]

(And thanks for the tip about not needing -U! I learned something. :-)

[piotr-dobrogost]

For some background see issue #2195 – pip install of a directory is super slow.
Problem with source directory containing sockets was raised in issue #2974 – Wheel command does not work if the directory contains a socket.

[pradyunsg]

Thanks to @chrahunt and @cjerdonek, there's a Python-3-only fix for this which skips copying socket files when copying the source directory. :tada: