Pipenv: Installing dependencies fails on dangling symlink

[dimexid]

Issue description

In new version of pipenv, 2020.5.28, we get following error while installing dependencies using command pipenv install --system --deploy from Pipfile.lock. We did not face this problem in previous version of pipenv.

Installing dependencies from Pipfile.lock (d6bfdd)…
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/cli/command.py", line 251, in install
    site_packages=state.site_packages
  File "/usr/local/lib/python3.7/dist-packages/pipenv/core.py", line 2065, in do_install
    keep_outdated=keep_outdated
  File "/usr/local/lib/python3.7/dist-packages/pipenv/core.py", line 1317, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.7/dist-packages/pipenv/core.py", line 847, in do_install_dependencies
    deps_list = list(lockfile.get_requirements(dev=dev, only=dev_only))
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 273, in get_requirements
    yield Requirement.from_pipfile(k, v)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2743, in from_pipfile
    r = FileRequirement.from_pipfile(name, pipfile)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1843, in from_pipfile
    arg_dict["setup_info"] = arg_dict["parsed_line"].setup_info
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 813, in setup_info
    self.setup_info = self.get_setup_info()
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 802, in get_setup_info
    setup_info = SetupInfo.from_ireq(self.ireq, subdir=self.subdirectory)
  File "/usr/local/lib/python3.7/dist-packages/pipenv/vendor/requirementslib/models/setup_info.py", line 1851, in from_ireq
    shutil.copytree(path, target)
  File "/usr/lib/python3.7/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/path/prefix/path/to/symlink/symlink_name.sh', '/tmp/reqlib-src99bh943o/path/to/symlink/symlink_name.sh', "[Errno 2] No such file or directory: '/path/prefix/path/to/symlink/symlink_name.sh'")]

The error is related to the following line of code:
https://github.com/pypa/pipenv/blob/06f5d7f58c789d815516008359a6dcc45f610e58/pipenv/vendor/requirementslib/models/setup_info.py#L1851

While installing dependencies, our symlink is dangling because it points out of directory visible to pipenv.

Expected result

Dangling symlinks inside package path might be configurable ignored

Actual result

Installing dependencies fails if dangling symlink sits somewhere inside package path

Steps to replicate

Our pipfile contents:

[[source]]
verify_ssl = true
name = "pypi"
url = "https://pypi.our.domain"

[packages]
our-package-name = {editable = true,index = "https://pypi.our.domain/",path = "."}

[dev-packages]

[requires]
python_version = "3.7"

Dangling symlink is located in "./scripts/symlink_name.sh" related to the path in [packages] section.
Command pipenv install --system --deploy causes described problem.

Solution

One of possible solutions might be to ignore dangling symlinks in shutil.copytree method ignore_dangling_symlinks=True (configurable perhaps). Regrettably, I am not sure if it can cause any problem elsewhere.

[frostming]

That argument is not available on Python 2.7, but IMO it seems more like a user error to me.

[dimexid]

I agree, it is also more likely user error from my point of view. But perhaps, it is way too strict to fail on copying dangling symlink which is not used anywhere.

[techalchemy]

This is resolved in #4302 -- ignoring the dangling symlinks isn't necessary, we will just copy them even though they are dangling. If that results in a breakage, then the user will have to resolve it.

Thanks for reporting and helping get this sorted out!