Pipenv: Request for `PIPENV_ONLY_USE_PYENV` environment variable.

[BrennanGit]

Is your feature request related to a problem? Please describe.

We are using a combination of linuxbrew, pipenv and pyenv on our linux agents in our CI system.
Unfortunately due to reasons, homebrew removed all installation options. This means that on Linux, all Pythons installed with brew do not have Tkinter support enabled.

It's an issue because, depending on what version is requested by the devs, they may or may not get a working Python and since brew usually has the most up to date releases, these are often the ones requested. This could be (and probably will be) worked around with much difficulty by creating our own Python formula on our own brew tap.

Describe the solution you'd like

I would love another environment variable that would force pipenv to only use pythons provided by pyenv. This would help make our CI system far more deterministic and predictable. Basically the opposite to PIPENV_DONT_USE_PYENV, maybe PIPENV_ONLY_USE_PYENV.

I could also see it being beneficial to those working on an older system like CentOS 6 who might only specify python 2 in their Pipfile. Currently the behaviour would be to use the system python (which I believe is 2.6) but I think most developers who write that in their Pipfile are working around the lack of version specifiers available for the Python version and mostly just want to use one of the most recent versions of Python 2.

Describe alternatives you've considered

For our particular CI system, we have resorted to refusing to allow many of our Python applications to be tested on our Linux agents. As mentioned above, we are considering creating our own Python formula with tk support hard coded.

Another option could be to provide a blacklist of Python executables that will not be considered available to pipenv like PIPENV_EXCLUDE_PYTHONS=/home/linuxbrew/.linuxbrew/bin/python:/home/linuxbrew/.linuxbrew/bin/python2:/home/linuxbrew/.linuxbrew/bin/python3.

Potentially this could also be a list of directories where any python executable found in within is removed from the list of available Python installations.

Additional context

We have everything working on our Mac agents since they have indeed hard coded tk support into the brew formula but it would still be nice to be able to say for sure that all Pythons available to pipenv in our CI system were created with pyenv.

This has been tested on both Ubuntu (18.04) and CentOS (7) but after looking at the linuxbrew python formula, it should be true of all linuxbrew installations.

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/bin/python3.7'

Python installations found:

• 3.6.7: /usr/bin/python3.6m
• 3.6.7: /usr/bin/python3.6
• 2.7.15: /usr/bin/python

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-54-generic',
 'platform_system': 'Linux',
 'platform_version': '#58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

• PATH
• PIPENV_VENV_IN_PROJECT
• LS_COLORS
• SSH_CONNECTION
• LESSCLOSE
• _
• LANG
• PYENV_ROOT
• PIPENV_MAX_DEPTH
• XDG_SESSION_ID
• USER
• PWD
• HOME
• SSH_CLIENT
• XDG_DATA_DIRS
• PIPENV_YES
• PIPENV_NOSPIN
• HOMEBREW_REPOSITORY
• SSH_TTY
• MAIL
• SHELL
• TERM
• SHLVL
• PYENV_SHELL
• HOMEBREW_PREFIX
• MANPATH
• LOGNAME
• XDG_RUNTIME_DIR
• PIPENV_COLORBLIND
• HOMEBREW_CELLAR
• INFOPATH
• LESSOPEN
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1
• PIPENV_MAX_DEPTH: 5
• PIPENV_YES: 1
• PIPENV_NOSPIN: 1
• PIPENV_COLORBLIND: 1

Debug–specific environment variables:

• PATH: /home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/tools:/home/REDACTED/.pyenv/shims:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/REDACTED

---

[frostming]

tldr; set PIPENV_PYTHON=$PYENV_ROOT/shims/python in your .bashrc to force pipenv to use pyenv python.

Tell me if it solves your problem.

[BrennanGit]

Thanks for the massively quick reply.

tldr; set PIPENV_PYTHON=$PYENV_ROOT/shims/python in your .bashrc to force pipenv to use pyenv python.

Tell me if it solves your problem.

Although this can be used to force pipenv to use a specific Python install, it does not really solve my problem. I'd still like the functionality provided where pyenv is used to install a python version that is not available, I'd just like a way to restrict which pythons could be considered usable by pipenv.

On a development machine, this would be a good solution (kind of equivalent to calling with --python /path/to/python I think?) but this is for a CI system and the goal is to allow developers to use whichever python version they require by specifying it in their Pipfile with the agent installing it with pyenv if it needs to.

[AlJohri]

@BrennanGit correct me if I'm wrong, but specifying PIPENV_PYTHON=$PYENV_ROOT/shims/python should technically allow for any pyenv installed python, depending on which is currently active, right? it's basically PIPENV_ONLY_USE_PYENV?

[BrennanGit]

Yes this would allow you to force pipenv to use the currently active python from pyenv though this is not the behaviour I'm after. I want to be able to use pipenv+pyenv in it's normal (extremely useful) way but without worrying about other python installations (sometimes with incompatible compilation options) cluttering the system.

Here's an example:

• A user requests python 3.7 in their Pipfile and sends it off to the CI system.
• The CI machine has python 3.7 installed via brew (with broken Tkinter on Linux) and other lesser pythons installed with pyenv. We could uninstall this python but it is a dependency for many other packages.

What happens now:

• pipenv selects the broken python
• matplotlib fails

With PIPENV_ONLY_USE_PYENV:

• pipenv looks for python 3.7 but ignores any outside PYENV_ROOT
• pipenv gets pyenv to compile the right python
• we get a working version, compiled from source in a predictable, deterministic way

It would be very nice to not have to worry about all the weird system/package manager pythons that may be installed as dependencies from other packages. It just gives us a way to ensure pipenv reliably does the same thing which is very important for continuous integration.

[BrennanGit]

Hi guys, just hoping maybe you could change the label from "question" to "enhancement" so this doesn't get lost. Once I have a little more time, I'd be happy to implement this myself and do a PR.

I'd just like to check that this is functionality that is understood and wanted by anyone other than myself.

[atomic-mattie]

@BrennanGit I'm interested in what I think this functionality is just for development because:

• my Pipfile specifies python_version = 3.7
• PIPENV_PYTHON=$HOME/.pyenv/shims/python uses the currently-selected version (system), which isn't compatible
• I'd rather have pipenv search through my pyenv and find that I've installed 3.7.5
• when I upgrade my OS, any virtualenvs based on a system Python break, in my experience—pyenv Pythons are stable

[BrennanGit]

Hey @atomic-mattie, it does sound like your use case matches mine. The ability to restrict pipenv to use only pyenv-installed pythons would definitely help reproducability.

In the end we created a pipeline function in Jenkins that sets up a pipenv like this:

> parse python version from Pipfile
> pyenv install --list
> find best matching python with semver (same rules as pipenv uses)
> pyenv install -s $MATCHING
> pipenv install --python $(pyenv root)/versions/$MATCHING/bin/python" 

It's quite a rouandabout way of doing it but it forces our builds to be more repeatable, even across new machines and os updates.