Pipenv: Virtualenv needs to be re-created for minor Pyhton updates on Homebrew (a.k.a please use venv!)

[intgr]

Issue description

Python versions 3.3+ bundle a venv module for creating virtualenvs (python -m venv). The virtualenvs created by venv are tidier and supported by Python upstream itself.

There is also a major issue with virtualenv on macOS using Homebrew: it creates symlinks specific to the currently installed package version. This means every time Homebrew updates Python (including minor version updates), I need to recreate all virtualenvs. This gets annoying quickly. venv virtualenvs, however, are not affected by this issue.

Expected result

Virtualenv created with venv:

% python3 -m venv ./venv
% . ./venv/bin/activate
(venv) % tree -L 4 $VIRTUAL_ENV
/private/tmp/envtest/venv
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.7
│   ├── pip
│   ├── pip3
│   ├── pip3.7
│   ├── python -> python3
│   └── python3 -> /usr/local/bin/python3
├── include
├── lib
│   └── python3.7
│       └── site-packages
│           ├── __pycache__
│           ├── easy_install.py
│           ├── pip
│           ├── pip-19.0.3.dist-info
│           ├── pkg_resources
│           ├── setuptools
│           └── setuptools-40.8.0.dist-info
└── pyvenv.cfg

11 directories, 12 files

Actual result + Steps to replicate

 % cd /tmp/envtest
 % pipenv shell
Creating a virtualenv for this project…
Pipfile: /private/tmp/envtest/Pipfile
Using /usr/local/opt/python/bin/python3.7 (3.7.4) to create virtualenv…
⠼ Creating virtual environment...Already using interpreter /usr/local/opt/python/bin/python3.7
Using base prefix '/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx/bin/python3.7
Also creating executable in /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment!
Virtualenv location: /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx
Creating a Pipfile for this project…
Launching subshell in virtual environment…
 . /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx/bin/activate
%  . /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx/bin/activate
(envtest) % tree $VIRTUAL_ENV -L 3
/Users/$USER/.local/share/virtualenvs/envtest-pJJWjZNx
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.ps1
│   ├── activate.xsh
│   ├── activate_this.py
│   ├── easy_install
│   ├── easy_install-3.7
│   ├── pip
│   ├── pip3
│   ├── pip3.7
│   ├── python -> python3.7
│   ├── python-config
│   ├── python3 -> python3.7
│   ├── python3.7
│   └── wheel
├── include
│   └── python3.7m -> /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m
└── lib
    └── python3.7
        ├── LICENSE.txt -> /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/LICENSE.txt
        ├── __future__.py -> /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/__future__.py
        [...]             Note version-specific paths ^^^^^^^

This is what it looks like to use a broken virtualenv:

% pipenv shell
Warning: Your Pipfile requires python_version 3.7, but you are using None (/Users/m/.local/share/v/l/bin/python).
  $ pipenv --rm and rebuilding the virtual environment may resolve the issue.
  $ pipenv check will surely fail.
Launching subshell in virtual environment…
 . /Users/marti.raudsepp/.local/share/virtualenvs/lemur-N4Rvgxif/bin/activate
%  . /Users/marti.raudsepp/.local/share/virtualenvs/lemur-N4Rvgxif/bin/activate
(lemur) % python
dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/marti.raudsepp/.local/share/virtualenvs/lemur-N4Rvgxif/bin/python
  Reason: image not found
zsh: abort      python

(lemur) % cat $VIRTUAL_ENV/lib/python3.7/__future__.py
cat: /Users/marti.raudsepp/.local/share/virtualenvs/lemur-N4Rvgxif/lib/python3.7/__future__.py: No such file or directory
(lemur) % ls -la $VIRTUAL_ENV/lib/python3.7/__future__.py
lrwxr-xr-x 99 marti.raudsepp 14 Aug 15:14 /Users/marti.raudsepp/.local/share/virtualenvs/lemur-N4Rvgxif/lib/python3.7/__future__.py -> /usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/__future__.py

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.7'

Python installations found:

• 3.7.4: /Users/marti.raudsepp/.local/share/virtualenvs/envtest-pJJWjZNx/bin/python3
• 3.7.4: /usr/local/bin/python3
• 3.7.4: /usr/local/bin/python3.7m
• 3.6.1: /usr/local/bin/pypy3
• 2.7.16: /usr/local/bin/python
• 2.7.16: /usr/local/bin/pythonw
• 2.7.10: /usr/bin/python
• 2.7.10: /usr/bin/pythonw
• 2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT '
                     '2019; root:xnu-4903.271.2~2/RELEASE_X86_64',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

[intgr]

I found issue #15 now. But while that there is worded like a feature request, what I submitted is a significant annoyance, if not bug, so I think deserves to stay open until some solution is found.

[joshsleeper]

fyi, I _think_ this comment expresses the same issue as you effectively?

https://github.com/pypa/pipenv/issues/15#issuecomment-356407377

[intgr]

fyi, I _think_ this comment expresses the same issue as you effectively?

A little similar perhaps, but not the same. dhouck there is talking about major Python upgrades, such as 3.6 to 3.7.

I'm talking about minor Python patchlevel updates, such as 3.7.3 to 3.7.4 or even Homebrew package re-builds, like 3.7.4 to 3.7.4_1 (which is what I used as example in the original report).

[intgr]

All my envs were broken again by Homebrew update to Python 3.7.5 :(

This time I used a hack to re-create Pipenv's environments by actually using python's own venv, and so far things work well:

cd ~/.local/share/virtualenvs
for dir in *; do
  rm -rf $dir
  python3 -m venv $dir
done

(Following that you still need to pipenv install in every environment directory to reinstall its packages)

[RNCTX]

all of mine are broken right now.

but I don't have any virtualenvs in ~/, I install them locally to the folder they're started in, and virtualenv is still broken

[shayneoneill]

This is a masssssssive annoyance by the way. As a python dev , I've usually got 10-15 virtualenvs on the go at any time, and about every couple of months something in the dependency chain will upgrade python and blam, everythings broken.

[RNCTX]

Even though it's a nice interface, I gave up on pipenv because venv is the officially supported virtual environment method upstream. While I was at it, I stopped using the Homebrew python installations in favor of the official python distributions for OSX.

I recreate the convenient pipenv shell commands via bash aliases like....

alias djangodev='set -a; source ~/Google\ Drive/djangodev/.env; set +a; \ 
source ~/.djangodev/bin/activate && cd ~/Google\ Drive/djangodev'

[uranusjr]

I believe this is no longer an issue after virtualenv 20.