Pipenv: Homebrew installation broken on Python < 3.7

[amancevice]

This is probably related to #2296, but I'm currently unable to upgrade pipenv past 2018.6.25 using Homebrew.

I have my Python pinned at 3.6.5 with brew switch python 3.6.5_1 && brew pin python

Because of this I have to --build-from-source for any upgrade to a brew-installed package that depends on python as the bottles are built on Python 3.7.

After brew upgrade pipenv --build-from-source I get the following error:

pipenv --help
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.10.9/libexec/bin/pipenv", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/Cellar/pipenv/2018.10.9/libexec/lib/python3.6/site-packages/pkg_resources/__init__.py", line 26, in <module>
    import zipfile
ModuleNotFoundError: No module named 'zipfile'

Switching back to 2018.6.25 resolves the issue, but it'd be nice to get on the latest version.

[techalchemy]

this seems incredibly weird, it looks to me like your python version doesn't have zipfile support. Unfortunately this is almost certainly not a pipenv problem -- but I have no real way to test this :(

[amancevice]

I'll give you that it is definitely weird! I do have zipfile installed on 3.6, though:

ls /usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6
...
-rw-r--r--   1 amancevice admin  74K Mar 28  2018 zipfile.py

I think you might be able to reproduce with the following steps:

Install Python 3.6.5

brew update

# Rewind Homebrew
cd $(brew --repo homebrew/core)
git checkout 69b99b69ff3a9c686b9d9fb0102927c5dd2f373b ./Formula/python.rb

# Install Python 3.6.5
brew install python
brew switch python 3.6.5_1

# Revert Homebrew
git reset @
git checkout ./Formula/python.rb

Re-Install Pipenv

brew reinstall pipenv --build-from-source

[ebb-earl-co]

@amancevice I am having the same issue. However, even going back to 2018.6.25 didn't solve it for me:

$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/2c0bbfa297e2429cd6e080ad5231f3aa56ff4f65/Formula/pipenv.rb
######################################################################## 100.0%
Warning: pipenv 2018.10.9 is available and more recent than version 2018.6.25.
==> Downloading https://homebrew.bintray.com/bottles/pipenv-2018.6.25.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring pipenv-2018.6.25.high_sierra.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/pipenv/2018.6.25: 1,350 files, 18.8MB
$ pipenv install
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.6.25/libexec/bin/pipenv", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/Cellar/pipenv/2018.6.25/libexec/lib/python3.6/site-packages/pkg_resources/__init__.py", line 26, in <module>
    import zipfile
ModuleNotFoundError: No module named 'zipfile'

[amancevice]

@ebb-earl-co if you're luck it'll be as simple as:

brew switch pipenv 2018.6.25

Typing brew switch pipenv by itself will show you all the versions you have installed. These get cleaned up on brew cleanup so if you run a tight ship it's probably long gone.

If it's gone you'll have to rewind your Homebrew tap to when it was the latest version. You can do this using the same logic as above for Python, but you'll need to find the correct checksum by grepping the logs. I'll walk you through it:

# Update brew (it's a good idea to always do this when beginning work on Homebrew)
brew update

# Navigate to the core repository
cd $(brew --repo homebrew/core)

# Search the log for checksum at `pipenv 2018.6.25`
git log --oneline | grep 'pipenv 2018.6.25'

# The checksum snippet is `9e1733c5c`

# Checkout the pipenv formula at this point in history
git checkout 9e1733c5c ./Formula/pipenv.rb

# Try installing the formula with
brew install pipenv

# If that doesn't work it might be because the package was bottled for a different Python version
# Try this:
brew install pipenv --build-from-source

# Fix your homebrew tap
git reset @ ./Formula/pipenv.rb
git checkout ./Formula/pipenv.rb

Be sure to pin your pipenv to prevent unwanted upgrades with brew pin pipenv.

[ebb-earl-co]

@amancevice thank you very much for the tips! Unfortunately, neither brew install pipenv nor brew install pipenv --build-from-source solved the ModuleNotFoundError ☹️

[amancevice]

On second look, it could be the post_install step that's causing issues.

For reference, here is the code:

def post_install
  lib_python_path = Pathname.glob(libexec/"lib/python*").first
  lib_python_path.each_child do |f|
    next unless f.symlink?
    realpath = f.realpath
    rm f
    ln_s realpath, f
  end
  inreplace lib_python_path/"orig-prefix.txt",
            Formula["python"].opt_prefix, Formula["python"].prefix.realpath
end

Commenting this code out (or simply changing the name of the method) seems to at least get pipenv --help to complete.

[amancevice]

OK, I've isolated the problem to this line in post_install:

inreplace lib_python_path/"orig-prefix.txt",
          Formula["python"].opt_prefix, Formula["python"].prefix.realpath

If I understand correctly, this line is replacing text in the file lib_python_path/"orig-prefix.txt"; specifically its replacing the value of Formula["python"].opt_prefix with the value of Formula["python"].prefix.realpath.

The problem is that Formula["python"].prefix.realpath points to 3.7.0 (or whatever the current version of the python formula is) and not my linked 3.6.5.

[amancevice]

Opened a PR to resolve the issue: https://github.com/Homebrew/homebrew-core/pull/32868

[amancevice]

@techalchemy it might help if you or some other pipenv maintainer weighed-in on the above PR. I think its reasonable to offer a formula that works on older Python Homebrew installations.

[uranusjr]

I’m probably missing something, but what do you mean by “older Python Homebrew installations”? I am not very familiar with Homebrew internals, but I assume the patch to Homebrew would be applied when users upgrade the formula. If that’s the case, Python installations shouldn’t be a factor here.

[amancevice]

I could be barking up the wrong tree here — I'm assuming that the maintainers of this project have some stake in the official Homebrew formula for pipenv, but that may be incorrect.

With Homebrew, your global formulae are kept up to date as a collection using _taps_, which are just git repos. When you brew update you are essentially pulling down changes for your taps, chiefly the core tap.

When you upgrade software with brew, the old installation sticks around until you brew cleanup. Switching between versions is relatively easy using brew switch. You can also brew pin formulae which prevents the installation from being upgraded.

In my case (and it sounds like I'm not the only one), I've pinned my Python installation to 3.6.5. But the post_install stage in the formula for pipenv there is some code that assumes the path to your Python installation is the same as the path to the _formula_ for Python. This means that brew install pipenv --build-from-source completes successfully but the software is left in a broken state.

I'm not a Homebrew expert and I can't say I really understand why the post_install stage is necessary in this formula — only that the way its written now it will only work if the version of Python users have installed matches the version of the Formula in their tap.

You can experiment with the Homebrew runtime using brew irb.

Get the formula for Python:

py = Formula["python"]

Show the path prefix that the post_install stage is going to replace:

py.opt_prefix.to_s
# => "/usr/local/opt/python"

Show the path prefix for the formula

py.prefix.to_s
# => "/usr/local/Cellar/python/3.7.0"

Show the path prefix of the _locally linked_ version of the formula

py.prefix(py.linked_version).to_s
# => "/usr/local/Cellar/python/3.6.5_1"

As you can see for me, #prefix and #prefix(#linked_version) are different.

I hope that helps explain things.

[uranusjr]

I don’t believe any of the maintainers contribute to the Homebrew formula. We try to fix things when they are broken, but none of us have much more stake in the formula than, say, you do.

I do have some personal experience with Homebrew, and your walkthrough does help explain things a lot. Thanks! Based on the above information, I feel your patch to the post-install hook makes sense. It makes things better in certain situations (i.e. yours), and does not break existing situations. Unfortunately, we don’t really have a say on this topic; you’ll need to convince the Homebrew maintainers to get it in :(

[uranusjr]

Closing since there isn’t really anything to track in this repo. Feel free to reopen (or raise a new issue) if cooperation from us is needed to resolve this issue.

[Samox]

Hello guys I had to rollback from python 3.7 to 3.6.4_4 (in order to use Celery).

I use these commands: brew install python 3.6.4_4 and brew switch python 3.6.4_4. Then pipenv stopped working because it could not find python 3.7. obviously. So I tried to reinstall pipenv and faced the same error ModuleNotFoundError: No module named 'zipfile'.

Do you know a way to install a proper python 3.6 environnement today ?
Thanks ! Merry Christmas !

[amancevice]

I ended up taking someone's advice and started using pyenv. The way things work now, my brew-installed Python exists to support Homebrew tools only -- it's completely isolated from the rest of my environment. In development I can choose which version of Python I want to use on a per-project basis (with global defaults). It was mildly annoying to change my workflow but the pain was minimal and I do think this is a superior configuration.

[Samox]

Thanks @amancevice :) I finally managed to make it work with pipenv. For documentation here are the steps:

• brew uninstall pipenv
• brew install pipenv
• brew install python 3.6.4_4
• Create a symlink ln -s /usr/local/Cellar/python/3.6.4_4/bin/python3 /usr/local/opt/python/bin/python3.6 because pipenv will ttry to find it here...
• now use the destructive command pipenv shell --python /usr/local/Cellar/python/3.6.4_4/bin/python3 a new virtualenv will be created
• You can now use pipenv cli pipenv shell or pipenv install with the python 3.6 version

@amancevice In my case I did not have to use pyenv. The only annoying thing is that you need to find where brew will install python distribution.