Pipenv: Install of a wheel from URL with pipenv

[nwh]

Hello, I am attempting to install PyTorch into an environment with pipenv.

See: http://pytorch.org/

System:

• macOS
• Python 3.6 (installed via Brew)
• CUDA: none

pipenv --three
pipenv install numpy
pipenv install http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl

Issue:

Installing http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl...
Collecting torch==0.2.0.post3 from http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl
  Downloading http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl (5.7MB)
Collecting pyyaml (from torch==0.2.0.post3)
Requirement already satisfied: numpy in /Users/nwh/.local/share/virtualenvs/proj2-d5yuf5ok/lib/python3.6/site-packages (from torch==0.2.0.post3)
Installing collected packages: pyyaml, torch
Successfully installed pyyaml-3.12 torch-0.2.0.post3

Adding http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl to Pipfile's [packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 936, in install
    do_lock(no_hashes=no_hashes)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 491, in do_lock
    results = resolve_deps(deps, sources=project.sources)
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 49, in resolve_deps
    for result in r.resolve():
  File "/usr/local/lib/python3.6/site-packages/piptools/resolver.py", line 94, in resolve
    self.their_constraints))
  File "/usr/local/lib/python3.6/site-packages/piptools/resolver.py", line 131, in check_constraints
    raise UnsupportedConstraint(msg, constraint)
piptools.exceptions.UnsupportedConstraint: pip-compile does not support URLs as packages, unless they are editable. Perhaps add -e option? (constraint was: torch==0.2.0.post3 from http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx-10-7-x86-64.whl)

Question: Does pipenv support wheel installs given a URL?

This alternative appears to work:

pipenv shell
pip install http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl

[kennethreitz42]

try -e

[nwh]

Here's what I get:

$ pipenv install -e http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl
Installing -e http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl...
⠴
Error:  An error occurred while installing -e http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl!
http://download.pytorch.org/whl/torch-0.2.0.post3-cp36-cp36m-macosx_10_7_x86_64.whl should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

[kennethreitz42]

thank you

[kennethreitz42]

we do not currently support this behavior

[kennethreitz42]

this was just added in master!

[kennethreitz42]

Wasn't easy, but it works!

[kennethreitz42]

this is now released.

[bosr]

Hello, using pipenv-7.3.0, I have the following error when trying to install that package with -e

$ pipenv install -e http://download.pytorch.org/whl/torch-0.2.0.post3-cp35-cp35m-macosx_10_7_x86_64.whl
Installing -e https://download.pytorch.org/whl/torch-0.2.0.post3-cp35-cp35m-macosx_10_7_x86_64.whl…
⠇
Error:  An error occurred while installing -e https://download.pytorch.org/whl/torch-0.2.0.post3-cp35-cp35m-macosx_10_7_x86_64.whl!
https://download.pytorch.org/whl/torch-0.2.0.post3-cp35-cp35m-macosx_10_7_x86_64.whl should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

Thanks!

[kennethreitz42]

I wouldn't do -e with a wheel.

[kennethreitz42]

$ pipenv install https://download.pytorch.org/whl/torch-0.2.0.post3-cp35-cp35m-macosx_10_7_x86_64.whl

[remram44]

I'm using this but sooner or later, running lock, pipenv is going to call egg_info on torch and that's going to fail. I still can't use pipenv (or pip-compile for that matter) with torch.

[techalchemy]

@remram44 2 questions:

1. Are you experiencing an issue currently if you pipenv install <some torch wheel>?
2. If pipenv were directly calling setup.py egg_info, why would it be problematic?

As a sidenote, there is a reason why we distinguish between editable and non-editable installs. Non-editable URLs are not deeply resolved, which means egg_info won't get called by running pipenv lock. These are explicitly excluded from being passed to the resolver.

[remram44]

pipenv install http://... works, but then when I install a package that depends on it (from a git+https:// source), apparently python setup.py egg_info is called and fails. Why it fails I'm not sure, but it still prevents me from using pipenv.

The package I install from a URL is pytorch, from a binary wheel (http://download.pytorch.org/whl/torch-0.3.0.post4-cp36-cp36m-macosx_10_7_x86_64.whl).

I'm not sure which setup.py it's trying to call btw, since the binary wheel doesn't have any.

[techalchemy]

@remram44 if you install a package that has a dependency which is unsatisfied, it will try to get it from PyPI, For most ML packages they are then built from sdists which means calling setup.py install. You already encountered this problem which is why you are using a wheel in the first place.

This is not a pipenv problem, it's a packaging problem in general. If you try to install something that has a dependency you can't build, it's going to fail no matter which of the various pip-related installers you try. The simple fact is if your system requires prebuilt wheels, you will need to install those.

[remram44]

The dependency is satisfied, installing the wheel with pip then the other thing works just fine. This is not an sdist but a wheel. I am using a wheel because it is the only way this is distributed (no sdist on http://pytorch.org/).

I do believe there is a pipenv bug here, because I see a traceback, and it looks like pipenv tried to execute some setup.py that might not exist at all.

[techalchemy]

@remram44 whatever bug you are encountering is not related to this issue. Please open another issue and include the traceback and all other relevant information.

[TotempaaltJ]

1. Are you experiencing an issue currently if you pipenv install <some torch wheel>?

I am! 🙋🏼‍♂️ Running the install _itself_ works with an HTTPS URL: the package _and its dependencies_ are installed. But, the dependencies never show up in Pipenv.lock, which means that any subsequent install of the application fails.

Is this related?

[techalchemy]

@TotempaaltJ it is! We have some issues installing remote wheels. Question: when it lands in your pipfile, what does it say?

[TotempaaltJ]

Here's the relevant line in [packages]:

"7abf3e1" = {file = "file://.../dependency/dist/test_dep-1.0-py3-none-any.whl"}

See also: https://github.com/TotempaaltJ/minimal-pipenv-wheel-fail

[rooom13]

So how can we properly install Pytorch with pipenv?