Pipenv: Support per-requirements overrides (--install-option and --global-option flags)

[numshub]

The problem

Since version 7.0 pip allows the usage of --global-option and --install-option individually for each requirement in requirements.txt (https://pip.pypa.io/en/stable/reference/pip_install/#per-requirement-overrides).

It is in principle possible to use environmental variables (PIP_INSTALL_OPTION and PIP_GLOBAL_OPTION), but they are propagated to all dependencies usually ending up in a failed install.

Required feature

Honor the install-option and global-option fields in Pipfile.

Due to the way this is implemented in pip, several space-separated arguments should be tranlasted into multiple --install-option / --global-option flags.

Example

[Pipfile]
...
apackage={version = "*", install-option = "--arg1 --arg2"}
...

[requirements.txt]
...
apackage --install-option="--arg1" --install-option="--arg2"
...

[techalchemy]

It would be helpful to see a use case here, I don't think any of us are eager to make fundamental changes to any parts of the spec. Since you are proposing a change to the pipfile spec you should file an issue against https://github.com/pypa/pipfile

For documentation purposes I'd be curious about the use case, but I'm probably going to close this and encourage you to file this against the pipfile repo

[numshub]

Thanks for the feedback.

I run into this issue while installing dd.
It is a typical example of a package that can optionally install cython extensions by passing a flag to pip (--fetch in this specific case).

Providing the same flag directly on the command line or as an environmental variable will fail as it propagates the flag to all dependencies (as documented here).

I will file an issue on https://github.com/pypa/pipfile as suggested.

[bjarchi]

I'm not sure this is the same issue but I think it is, so here's another (possible) use case that doesn't seem to be supported with current syntax I can find documented:

The cleanest way I have found to use system-installed pyqt(5) in a venv is to install vext.pyqt5, which basically installs shims to use the system copy of pyqt5 in the virtual environment. For some reason this package does not play nice when installed from wheel - and I agree this is probably a bug in the package that should be addresssed - so in a classic venv workflow you would install with:

pip install --no-binary vext.pyqt5 vext.pyqt5

I have confirmed that using this variant of the install command works (on previous pip versions it would apparently have been --no-use-wheel). Unfortunately I cannot find any way to specify this package in my Pipfile and have it get installed when I spin up an environment. I've tried things along the line of this:
"vext.pyqt5" = "*"
and
"vext.pyqt5" = {version = "*", install-option = "--no-binary vext.pyqt5"}

I have not had any success with this in my Pipfile when I try to initially create an environment using pipenv install. I get a warning about the failed package installation, a notice that it's retrying, this appears to succeed (the command finishes w/o error), and the package is listed as installed but non-functional.

Interestingly: if I leave this out of my Pipfile initially, and then after installing the rest of the environment I use pipenv install vext.pyqt5 to install the package, that command reports a (test?) error but I get an entry added to my Pipfile and a functional package. This can be verified by, e.g., import pyqtgraph in my context.

I recognize that in this case it may be addressable by correcting some issue in the vext.pyqt5 package, but I still think it's another argument for adding this kind of argument to the Pipfile spec - and I will add this comment to an issue on the Pipfile repo if/when that happens.

Edit: seems that the vext author is aware of this as a (non)-problem since 2016 - https://github.com/stuaxo/vext/issues/37 - maybe I should open/reopen the issue there since it now is an issue for pipenv initialization?

I didn't want to paste the full error message above because it's long, but here's the relevant part from pipenv install vext.pyqt5:

...
  Requirement already satisfied: ruamel.yaml>=0.11.10 in /usr/local/Cellar/pipenv/2018.5.18/libexec/lib/python3.6/site-packages (from vext>=0.5.21) (0.15.37)
  Checking .pth file support in build/bdist.macosx-10.12-x86_64/wheel/
  /Users/jbarchi/.local/share/virtualenvs/spectros_algo-60Hjheru/bin/python3.6 -E -c pass
  TEST FAILED: build/bdist.macosx-10.12-x86_64/wheel/ does NOT support .pth files
  error: bad install directory or PYTHONPATH

  You are attempting to install a package to a directory that is not
  on PYTHONPATH and which Python does not read ".pth" files from.  The
  installation directory you specified (via --install-dir, --prefix, or
  the distutils default setting) was:

      build/bdist.macosx-10.12-x86_64/wheel/

  and your PYTHONPATH environment variable currently contains:

      '<snip>'

  Here are some of your options for correcting the problem:
...

[bjarchi]

@techalchemy Any thoughts on this use case? I realize it is an issue with the package, but in my experience there are a number of useful packages out there that require specific pip options to install cleanly so I think there's some utility in having this ability.

Also, should I file a separate issue for the fact that the package installs functionally (albeit with apparent error) when using pipenv install package but not when using pipenv install to initially create the environment?

[uranusjr]

Personally I feel at least no-binary should be supported because it is needed quite often. Maybe we can make the option better (use TOML mapping instead of specifying raw pip arguments), but we should have something.

[techalchemy]

Do environment variables not work here? PIP_NO_BINARY=:vext:

[uranusjr]

Aw, yes, now I think of it, PIP_NO_BINARY already covered that use case. We (somebody) should compile a list of possible install options, and see if there are alternatives for them (and if not, discuss what we can do to cover the use case). This would work well as a page in the documentation, too.

[dmitriyshashkin]

Is there a way to put environment variables into the pipfile? Specifying them manually doesn't seem like a good solution. It would be extremely easy to forget what set of variables were used and end up with a different install from the same pipfile

[uranusjr]

@dmitriyshashkin Good point. Again, we’d probably need to plan this in a more comprehensive way.

[bjarchi]

EDIT: Original comment preserved below, but I no longer think the vext.pyqt5 issue is related to installing with --no-binary vs. not. Reasons:

• 0.5.21 still errors on pipenv install even with `PIP_NO_BINARY=vext.pyqt5 [or :vext.pyqt5:]
• 0.7.0 installs OK from within pipenv shell without PIP_NO_BINARY (and works) but does not work (and errors) when installed via pipenv install
• 0.7.0 apparently installs ok via pipenv install --sequential with the package late in my dependency spec
  
  
  
  • But still does not function
  
  
• 0.7.0 installs and functions after pipenv uninstall vext.pyqt5 && pipenv install vext.pyqt5 - but only when these commands are run from within the pipenv shell
  
  
  
  • Error on install when pipenv install vext.pyqt5 is run from main shell (outside venv subshell) after environment is initialized.
  
  

My best analysis at this point suggests two issues: vext.pyqt5 may not be properly declaring dependencies (hence error when not installed using --sequential), and it seems to be doing something that requires the environment to be active in order to install correctly.

At this point I'm not sure if this merits its own issue. The whole point of the vext system is to selectively break the barrier between venv and system for specific packages, pipenv/Pipfile seem to be the next evolutionary step in environment management, and (this) vext package doesn't play well with the normal install procedures.

At this point my options seem to be either --site-packages + PIP_IGNORE_INSTALLED or keeping template Pipfile/Pipfile.lock in VCS and instructing users to copy them locally before the two-stage install. I would appreciate advice as to whether I should open an issue to track this specific behavior.

-- Original comment--

Do environment variables not work here? PIP_NO_BINARY=:vext:

I admit I was not aware of this option, but unfortunately I just tried it and it does not appear to work in this case.

Snippet (using fish)

$ set -x PIP_NO_BINARY ":vext.pyqt5:"
$ pipenv install
...
Updated Pipfile.lock (e7a3de)!
Installing dependencies from Pipfile.lock (e7a3de)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 20/20 — 00:00:17
An error occurred while installing vext.pyqt5==0.7.0! Will try again.
Installing initially–failed dependencies…
Success installing vext.pyqt5==0.7.0! 0/1 — 00:00:00
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run the following:
...

This is the same output I got before, and while it appears to succeed on the second try it does not result in a functionally installed package (as evidenced by >>>import pyqtgraph failing with a failure to find Qt).

Unfortunately vext.pyqt5 was recently bumped from 0.5.21 to 0.7.0, and with 0.7.0 it doesn't seem like anything I do results in a functional interface anymore - although it no longer has the error on binary install? It may be that vext is more trouble than it's worth.

[jcushman]

I just wanted to add another example of why it's very useful to specify the equivalent of --no-binary in the pipfile.

There's an issue right now with the package "hiredis" where a new maintainer joined the project and added binary wheels for a three-year-old release, so the pipfile.lock hashes stopped matching. This is all probably fine and good (and I don't mean to criticize that particular maintainer for being helpful!), but for a while it was unclear whether the binary wheels were legitimate -- and it's still not 100% clear, as the original maintainer is asking people to confirm that the wheels match the source.

So while that issue sorts itself out, I would like to keep the original hashes and make sure that hiredis installs from source across all deployments. The pipfile is by far the preferable place to make sure that happens -- doing it otherwise basically requires wrapping pipenv in a custom script.

[gabrielcs]

One more use case: to install LightGBM with GPU support. Following @numshub's syntax it would look like:
lightgbm={version="*", install-option="--gpu"}

[tdbgamer]

Do environment variables not work here? PIP_NO_BINARY=:vext:

So it's okay for pipenv install to blow up and not work unless they know to set PIP_NO_BINARY every time they install a project? This makes pipenv a lot less useful to me. The big selling point was that you can just open a project and run pipenv install to get everything working.

Say what you will about javascript, but I've never seen anyone have to do NPM_NO_BINARY=:all: NPM_GPU=1 NPM_FETCH=1 npm install.

As a side node, --no-binary is pretty important when installing scientific libraries as well. Pip supports such a wide range of options that it makes sense to me that there would be a way to pass arbitrary options to pip.

[techalchemy]

Making a post asking us about whether it’s ok for pipenv to blow up is not very productive. Obviously if a command fails instead of succeeds, as a user, you just had a negative experience.

On the other hand, someone needs to actually think about the solution, approach, and implementation and propose that in an enhancement proposal PR to the peeps folder at the root of the repository. If you want my opinion, I personally find the UX aspect persuasive enough to be willing to review implementation details. Not sure if Kenneth wil agree that environment variables are as burdensome as you say.

Consider this: anything we implement now, we will be stuck with forever. We err on the side of not implementing things that have solutions or workarounds unless the UI/UX is sufficiently bad. As a maintainer, I have to pay for poorly thought out implementations in maintenance overhead later

[tdbgamer]

I intended it more as a rhetorical question to demonstrate that environment variables aren't a viable solution for these issues. If it's required to be set to build successfully, it should be somehow represented in the configuration.

[rgov]

I did not see an issue in the Pipfile repo to hash out the spec, so I filed: https://github.com/pypa/pipfile/issues/115

My use case: The Protobuf package contains a C++ extension which has a crashing bug. I could work around it by building from master with the --cpp_implementation build option.

[PatTheMav]

Here's another use case that I can't specify in the Pipfile:

To make uWSGI work with PyPy, both need to use the same libraries. uWSGI will use system libraries, whereas portable PyPy will use its own (e.g. OpenSSL 1.1.1 vs the system's OpenSSL 1.1.0j). As PyPy is loaded as a library into the uWSGI process, this will fail due to library mismatches.

The easiest fix is to run UWSGI_PROFILE=pypyonly pip install uwsgi --no-binary uwsgi which makes pip install uWSGI from sources, using the pypyonly profile (which excludes additional libraries). The Wheel variant is always built using another profile, so UWSGI_PROFILE is not applied.

[cdhagmann]

Here's another use case:
pip install python-ldap \ --global-option=build_ext \ --global-option="-I$(xcrun --show-sdk-path)/usr/include/sasl"

This works to install python-ldap on MacOS

[crifan]

my use case:

pip3 install pycurl==7.43.0.1 --global-option="--with-nss" --upgrade --no-cache-dir

so using pipenv to install, expect support something like this:

PIP_INSTALL_OPTION="--global-option='--with-nss' --upgrade --no-cache-dir" pipenv install pycurl==7.43.0.1

[felix-ht]

my usecase to install gdal:
pip install GDAL==$(gdal-config --version) --global-option=build_ext --global-option=-I/usr/include/gdal

[thehesiod]

btw, perhaps one way around this is by loading .env during pipenv install ? Then you can specify things like PIP_NO_BINARY, however that would not be module specific

[clusterrun]

my usecase to install dulwich

It allows skipping the installation of C bindings by using the --pure option:
pip install dulwich --global-option="--pure"

[mpbrown]

My use case is installing django-compressor

On Windows, django-compressor's dependencies throw Visual C++ extension errors even though the required extensions are installed. Issue: rcssmin and rjsmin do not install These install options allow it to install without c extensions.

pip install rcssmin --install-option="--without-c-extensions"
pip install rjsmin --install-option="--without-c-extensions"

Without install options for pipenv, we are forced to use a requirements.txt file. Thus we are currently using pipenv only for managing a virtual environment rather than managing packages, making our install documentation more complicated than pipenv aspires to be.

[polidore]

just went through this installing lightgbm with gpu support.