Pipenv: Provide better error message if the project’s virtual environment is broken

[slhck]

Was: "Your Pipfile requires python_version 3.6, but you are using None" and various errors

---

I run pipenv, installed via brew install pipenv on macOS 10.12. I've installed python from Homebrew (3.6.5), as well as python@2.

When I open pipenv shell, I get this output:

➜ pipenv shell
Warning: Your Pipfile requires python_version 3.6, but you are using None (/Users/werner/.local/share/v/b/bin/python).
  $ pipenv check will surely fail.
Spawning environment shell (/bin/zsh). Use 'exit' to leave.

When I run pipenv check, I get:

Checking PEP 508 requirements…
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==11.9.0', 'console_scripts', 'pipenv')()
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/cli.py", line 642, in check
    three=three, python=python, system=system, unused=unused, args=args
  File "/usr/local/Cellar/pipenv/11.9.0_1/libexec/lib/python3.6/site-packages/pipenv/core.py", line 2275, in do_check
    results = simplejson.loads(c.out)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I cannot install anything either:

$ pipenv install pymysql
Installing pymysql…
⠋
Error:  An error occurred while installing pymysql!
dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/werner/.local/share/virtualenvs/data-analysis-xEhMcwIZ/bin/python3.6
  Reason: image not found

Please run $ python -m pipenv.help, and paste the results here.

/usr/local/opt/python@2/bin/python2.7: No module named pipenv

When I run python3 -m pipenv.help:

/usr/local/opt/python/bin/python3.6: Error while finding module specification for 'pipenv.help' (ModuleNotFoundError: No module named 'pipenv')

Expected result

I expected to be dropped into the pipenv shell without errors (or warnings, that is).

Actual result

See the errors mentioned above.

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile):

I installed pipenv via Homebrew, then ran:

pipenv --three
pipenv install matplotlib seaborn pandas scipy numpy SQLAlchemy
pipenv shell

A few days later my Python 3 was updated via Homebrew, if I remember correctly. Not sure if that could cause any problems.

[slhck]

The Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
scipy = "*"
matplotlib = "*"
seaborn = "*"
pandas = "*"
sqlalchemy = "*"

[dev-packages]

[requires]
python_version = "3.6"

[slhck]

I did a pipenv --rm and pipenv install, now everything seems to work again.

[uranusjr]

Some debugging tips for future reference: The image not found error usually comes from a broken installation, i.e. your virtualenv was not created correctly, or broken by some operation. Re-creating (or re-installing if it’s the global Python that’s broken) is the best solution.

[slhck]

Thanks! I haven't worked with virtualenvs much – my assumption is that the Homebrew python update broke it. From what I see now there's a python executable linked from the virtualenv to the system python. That might explain the errors…

[uranusjr]

@slhck Homebrew tends to do that. My suggestion is to not use Homebrew Pythons to create virtualenvs you intend to keep them long-term.

[rca]

Another scenario where this comes up: you created a pipenv with a Python installation you have since removed / upgraded / etc.

I ran into this because I had a Python3.6 installation from python.org and then moved over to using Python3.6.5 from Homebrew. After removing the python.org installation from my system entirely, I encountered this error. No big deal if you know why: the virtualenv's python is pointing to a python installation that no longer exists.

I presume (have not run into this yet myself) that if this problem happens more with Homebrew than other installations, it's because Python is more frequently upgraded running something like brew upgrade (analagous to apt-get upgrade) and it swaps out the python installation, effectively leaving the virtualenv orphaned.

Some of this is conjecture, but it's my best attempt at understanding the problem.

Thanks!

[uranusjr]

Homebrew is more problematic because this happens for every point release. With python.org installers (and apt, for this matter) this generally happens only every minor release (e.g. 3.5 -> 3.6) because it tries to maintain binary compatibility between point releases, which makes problems happen less frequently and more obvious to debug. Homebrew also does not try to keep multiple versions of a software around (it does not delete it when you upgrade, but will feel free to clean it up when it sees fit), so you need to take extra care to keep (for example) 3.5 environments working after 3.6 is out. With python.org installers (and apt, again) you can easily have 3.5 and 3.6 installations exist side-by-side.

Note: It is possible achieve the same side-by-side installation in Homebrew, but the procedure is much more awkward, and does not transition well from the default Python installation if you don’t start with the correct setup. This is a deficiency in Homebrew’s design, they acknowledge this, but it is really difficult to do (I don’t blame them to leave things as-is).

[rca]

Got it, thank you for the additional context. What you describe makes sense and I'll definitely keep an eye out.

With respect to the issue revolving around the error messaging; would it be appropriate for getting some sort of friendly message saying something along the lines of, "pipenv --rm && pipenv install may resolve the problem" ?

[uranusjr]

Yes, I would imagine it is reasonable to hint the user the Python interpreter they are using is broken if python_version() returns None (or maybe we can raise more fine-grained exceptions for various errors, but that can be done incrementally). We can then suggest how they can rebuild the environment (with the commands you suggested).

[sinscary]

@uranusjr is this issue still open to working on? Can I take this up?

[uranusjr]

@sinscary Please do! In general you are always welcomed to take on anything labelled as “good first issue”.

[sinscary]

@uranusjr can you assign it to me then, I will start working on this.

[uranusjr]

@sinscary GitHub does not allow us to assign an arbitrary user (I never understood its logic), but I’ll note this internally. Thanks for the hard work in advance!

[iuriguilherme]

For duck.com or google searchers reference, this happened to my pipenv installs when I dist-upgraded my debian from stretch to buster.

Since it's the python from the system that has been tampered with, I endorse the solution of pipenv --rm && pipenv install.

If you just installed the debian package from apt install pipenv or in any way changed major things in the system, try to use other pythons you have. For example:

$ python3 -m pipenv --rm
$ ~/.local/bin/python -m pipenv --rm

And so on.