Homebrew-core: Links to current Python are lost after Homebrew update

[orome]

Pior to my latest update of Homebrew (1.2.4) I had Python located, as indicated by brew --config, with a link to the latest Homebrew version:

Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin/python2.7

and both this and the macOS system-provided Python were visible to which -a.

Following the update, I've lost access to this Python (though it still exists, and is available as python2) and just the old system-supplied Python is indicated by listed by brew --config as

Python: /usr/bin/python

and by which -a.

Is this the intended behavior of the latest Homebrew? If not, is there a fix for restoring the access I once had to the latest Homebrew Python?

---

Note that I also get the following results (which means I can't use brew to fix itself):

$ brew link --overwrite --dry-run python2
Warning: Already linked: /usr/local/Cellar/python/2.7.13_1
To relink: brew unlink python && brew link python
$ brew link --overwrite --dry-run python
Warning: Already linked: /usr/local/Cellar/python/2.7.13_1
To relink: brew unlink python && brew link python
$ brew link --overwrite --dry-run python3
Warning: Already linked: /usr/local/Cellar/python3/3.6.2
To relink: brew unlink python3 && brew link python3

---

Naively I assume I can fix this "by hand" with

redacted

But I wonder, given that Homebrew has made such a profound change to the Python configuration, whether more is not going on (that such linking will break). After all, the latest update chose to remove this link when it was present.

[kevwil]

Following, as my "python" and "pip" are also missing.

[DomT4]

Is this the intended behavior of the latest Homebrew?

Yes. See brew info python:

==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.zshrc:
  export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Pip and setuptools have been installed. To update them
  pip2 install --upgrade pip setuptools

You can install Python packages with
  pip2 install <package>

They will install into the site-package directory
  /usr/local/lib/python2.7/site-packages

See: http://docs.brew.sh/Homebrew-and-Python.html

[xeron]

I can confirm that adding /opt/brew/opt/python/libexec/bin into $PATH fixes this issue for me. However I also missed this notes after the update and thought that it's some bug.

[ghazel]

This intended change does seem like a mistake to me. Python and pip used to work, and now they do not.

Can we have the old formula back?

[slhck]

This also got me very confused – seems like a quite big change.

Ideally, this page should (have) be(en) updated: http://docs.brew.sh/Homebrew-and-Python.html

[jbub]

Seems like a pretty huge breaking change

i was using VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python, where should it point now ?

[MikeMcQuaid]

This is an intentional change (https://github.com/Homebrew/homebrew-core/pull/14408) because for years people have been complaining about things that need to depends_on "python" or depends_on :python being installed without their explicit request which overrides their which python. I appreciate this is frustrating if you're in the opposite camp (you liked that behaviour) but there was no workaround for the prior group. Additionally, the way Python is being handled now is more consistent how we should be handling other formulae that shadow system versions.

Note from the discussion in https://github.com/Homebrew/homebrew-core/pull/14408 that the long-term plan is to have brew install python install Python 3 and Python 2 be a keg-only install with brew install python@2.

Ideally, this page should (have) be(en) updated: http://docs.brew.sh/Homebrew-and-Python.html

Done in https://github.com/Homebrew/brew/pull/2912. That said, you were told this by the caveats when Python was installed.

I can confirm that adding $(brew --prefix)/opt/python/libexec/bin into $PATH fixes this issue for me.

This is the correct workaround if you wish to have python and pip in your $PATH. It's a one-time change for you.

[MikeMcQuaid]

i was using VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python, where should it point now ?

/usr/local/bin/python2?

[jbub]

Thanks for the info, i feel like homebrew should prompt user for confirmation when upgrading formulas with such breaking changes, so that people avoid scenarios like these. I run brew update/upgrade everyday and i did not really spot any warning :) Anyway seems like this is a nice cleanup. Keep up the great work! Thank you

[ilovezfs]

@jbub that is a nice idea, but nothing in Homebrew is currently interactive, so instead we print caveats.

[jbub]

Yeah i mean its my responsibility to check for compatibility problems, its just i have never really had any problems like these so i never really checked the caveats, thats the nice thing of homebrew, its been really stable for years.

[ilovezfs]

so i never really checked the caveats,

Tsk, tsk :)

We're aware that they aren't always read, or are under-read, so we do make an effort to keep them as brief as possible to avoid having people tune out and never read them.

that's the nice thing of homebrew, its been really stable for years.

Thanks! We appreciate the kind words.

[orome]

And I assume that adding /usr/local/opt/python/libexec/bin to /private/etc/paths will also do the trick. Correct?

[ilovezfs]

@orome yes, or /etc/paths.d/foo

[orome]

FWIW, IMV, the mistake here is not in the change of configuration (which makes good sense) but in the way it was (not) communicated (which was quite inadequate).

Minimally there should have been a clear message added to the docs; but even better would have been a much more prominent message in the "caveats". I don't know whether I'm a typical user, but in my experience, these have been entirely ignorable (as the message itself states): in years of using Homebrew I've had to attend them perhaps once (or not at all). The implication and experience is that doing anything they suggest is mostly optional. But in a case where critical and frequently used tools have suddenly become inaccessible, a more prominent and more complete message is required.

[orome]

@ilovezfs

yes, or /etc/paths.d/foo

Except that that will put the Homebrew Python _after_ any system Python, which is not the same as the suggested modification to PATH or using /private/etc/path. No?

[DomT4]

The issue of framing caveats prominently enough is something that has been discussed a lot over the years without much agreement on how to advance there. Certainly if you upgrade not incredibly regularly you can pile up upgrades in a way that spotting individual caveats, perhaps highlighting breaking changes, becomes rather difficult.

[slhck]

It's all “would've” and “could've”, but certainly, if the Homebrew Python webpage had been updated at the same time, I would've been able to confirm that this behavior was intentional. The description on the website is now much better, and it all makes sense now, thanks!

Perhaps the formula caveat should've said:

Homebrew does not add a python executable to $PATH (anymore), so as not to override the macOS system python. Instead, this formula installs a python2 executable to #{HOMEBREW_PREFIX}/bin.

Then it would've been ultra-clear to me.

But isn't the real issue about changing existing behavior? It really depends on whether you:

1. already had the formula installed (in which case it broke things or at least introduced significant changes), and it got updated, or
2. you installed that particular formula the first time, or
3. it got installed as a side-effect

My assumption would be that the importance of showing such caveats decreases in that order; probably most people don't care in the latter case, and – of course – would be surprised if their system python suddenly changed.

[ilovezfs]

Let's try to focus on actual technical issues with the current setup please.

[MikeMcQuaid]

way it was (not) communicated (which was quite inadequate)

It was communicated in multiple ways:

• there was a pull request open detailing the change for a while
• brew update and brew upgrade showed a Python change
• the caveats changed
• the docs changed within 24h

I'm not really interested at this stage in doing a post-mortem on whether it was or wasn't communicated well enough or being told (repeatedly) about our failings as a project. A gentle reminder: Homebrew is run by volunteers and you use it for free. If you run a similarly scale package manager and you can point to your better implementation of handling changes like this: that would be helpful.

As for the change itself: people were complaining about how it was before, people will be complaining about how it is now. It's literally impossible to keep everyone happy or informed with a change like this. We did our best, next time we may do differently. If you want to help: consider watching the project and providing feedback like this on pull requests before they are merged.

Note this issue is being left open to avoid duplicate issues for a few days. I don't consider there anything actionable to do based on this thread.

[orome]

I'm not really interested at this stage in doing a post-mortem on whether it was or wasn't communicated well enough

I'm not sure what your objection to improving communications is. It's clear what the problem here is, so it's really not helpful to anyone that you don't ant to address it. Simply wording the "caveat" better would have done the trick ("Note that this is a change from earlier configurations. Previous users of Python should ... to restore the access to Homebrew Python they have had in the past.").

or being told (repeatedly) about our failings as a project

If that's how you choose to interpret feedback, perhaps monitoring issues is not a good use of your time.

[MikeMcQuaid]

Simply wording the "caveat" better would have done the trick

Given multiple people above have said they don't read the caveats: unfortunately, it wouldn't.

Pull requests to Homebrew are welcome, that's the best way to discuss an improvement.

If that's how you choose to interpret feedback, perhaps monitoring issues is not a good use of your time.

Just to repeat what you've said back at you: you're suggesting the lead maintainer of Homebrew stop reading Homebrew issues. Clearly that isn't going to happen.

[orome]

@MikeMcQuaid

Just to repeat what you've said back at you: you're suggesting the lead maintainer of Homebrew stop reading Homebrew issues.

I think you see the logic error there: there's another adjustment that could be made (to the premise).

Given multiple people above have said they don't read the caveats: unfortunately, it wouldn't.

I can only speak from my own experience. If the caveat had been worded in any of the ways that have been suggested, that would have made all the difference. And it's worth taking this to heart (again, IMV) as a general rule: if something as dramatic as access to installed tools has changed, it's worth the few words it takes to simply say that right in the caveat.

[MikeMcQuaid]

it's worth the few words it takes to simply say that right in the caveat.

Ok, to try and be even more explicit: we will review a pull request from you on this matter but we do not want any more comments or suggestions from you in this issue, thanks.

[orome]

@MikeMcQuaid : Can you point me to where this the relevant message is generated in the code?

[slhck]

@orome: https://github.com/Homebrew/homebrew-core/blob/e6b3e73c43ffa170cbd841dbe2bc38acf0e20623/Formula/python.rb#L353

(This is the latest commit as of posting this message. When editing, make sure to use the latest version of the file, and check if there is no open PR already.)

[micahculpepper]

Hey folks,

In my organization, we use Macs with brew-installed Python. We depend on brew shadowing the system Python in order for a lot of our internal Python scripts to work.

Based on the number of people that seem to be having problems with this change, it seems like it would make sense to keep the old way of doing things around, just not as the default way. I would like to see an install option or a brew link option to create the links the old way.

I'll see if I can come up with a PR to do this so we can have something more concrete to discuss, but does this sound like a reasonable option? I would prefer not to edit my users' bash profiles, or require them to do so, in order to get things to work the way they did yesterday.

Edit:

So, based on comments from @ilovezfs and @MikeMcQuaid, I will not be making a PR for this. The long-term solution for my use-case involves a more involved setup script for installing our scripts, that creates virtual environments that satisfy our requirements and then just link the executables into the user's path. Thank you for pointing out PEP 394 -- The "python" Command on Unix-Like Systems, that was especially helpful.

[ilovezfs]

Please do not post code "workarounds" here that will lead to non-standard installations.

[ilovezfs]

I'll see if I can come up with a PR to do this so we can have something more concrete to discuss, but does this sound like a reasonable option?

No, because this will trigger a source build. Please prepend $(brew --prefix)/opt/python/libexec/bin to your PATH if you need an unversioned Homebrew python executable.

[MikeMcQuaid]

I would prefer not to edit my users' bash profiles, or require them to do so, in order to get things to work the way they did yesterday.

You could also adjust your scripts to use python2 or even Homebrew's full path or use the system Python 2.

[ilovezfs]

You could also adjust your scripts to use python2

Also, note that this is the recommendation of PEP 394 -- The "python" Command on Unix-Like Systems.

[orome]

@ilovezfs :

Also, note that this is the recommendation of PEP 394 -- The "python" Command on Unix-Like Systems.

But doesn't the new formula also break this PEP without some user intervention, even on systems that had been following it?

The more general python command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the python2 command.

Following execution of this formula (at least for me and the others posting here) this is not what happens: python and python2 invoke different Pythons, until the path changes suggested in the caveat are made.

[ilovezfs]

No.

[therealmarv]

When you are quoting PEP 394 recommendation let me quote some more:

The more general python command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the python2 command

So with your installation method you are against the PEP 394 recommendation because python is not python2 (Homebrew) by default!

[orome]

@ilovezfs :

No

Can you elaborate? Following the update, unless I change my path, python runs a different Python than python2 (against PEP 394). Prior to the update python ran the same Python as python2 (as in PEP 394).

[ilovezfs]

I suggest you re-read the caveat.

[MikeMcQuaid]

@orome I quote myself above:

we do not want any more comments or suggestions from you in this issue, thanks.

As a result, you're getting blocked from the Homebrew organisation for now.

[MikeMcQuaid]

Another reminder to everyone here: I was strongly in favour of having brew install python install Python 3 entirely and was convinced to compromise. I'm now increasingly wondering what the point of compromise is (mostly) with people who don't seem to be willing to expend any work to make Homebrew better but instead continue to snark and 👎 at the people who spend a lot of their free time to provide you with software for free.

[kkirsche]

For me it's just a matter of finding a way to better communicate the change to individuals who have it installed. I don't think the issue is necessarily the change, but rather that when doing something like brew update and updating a large number of packages at once, this can easily get lost in the output.

This is what bothered me personally. It's not to say that the change itself is a problem, just that I wonder if there is a better way to communicate these kinds of changes. Not your fault at the end of the day that I missed it, just frustrating when things get missed

[MikeMcQuaid]

I wonder if there is a better way to communicate these kinds of changes.

I'm 100% sure there's a better way to communicate changes. The question is: what does the implementation look like and who is willing and able to implement it?

[leobalter]

This issue is another case I appreciate a good use of semver. So bad homebrew could not prevent me from that and I had to try many things until I found this issue.

It was disappointing to figure out what was happening.

It does not make much sense when the installed pip2 goes to the usual path but not python itself.

I really like to use homebrew, it's a necessary tool, but the communication can be improved on breaking changes.

[therealmarv]

How about a mail list for announcements and/or a twitter account when breaking changes appear?

Maybe sites like https://brew.sh/blog/ and https://twitter.com/machomebrew are too less used? But would like to see software related breaking changes on some other place.

Or is there even a command before doing brew upgrade to see what will change? <- Actually it would be a good to read all new info texts before doing a brew upgrade for the not so adventurous upgrader 🤔

[MikeMcQuaid]

but the communication can be improved on breaking changes.

Please, please stop just posting the same comment again and again, people. If you have something new and incredibly insightful to add that hasn't been said before. I've blocked one person and I don't want to but I will block more if I have to. Again:

The question is: what does the implementation look like and who is willing and able to implement it?

[DomT4]

The question is: what does the implementation look like and who is willing and able to implement it?

Something I was thinking about earlier in terms of ideas is in the case where there's multiple upgrades not dumping the caveats until everything has finished upgrading, but I'm not sure how many people that would benefit in this particular discussion.

[kkirsche]

In terms of implementation I like @DomT4's suggestion of dumping the caveats at the end, grouped by package.

Another thought (just trying to brainstorm ideas) would be to have a formula implement a variable that's last_compatible_version potentially and use that to compare against the version being upgraded from. Then we could do something like if installed_version < last_compatible_version we could then prompt the user with a dialogue or something that explains that you are about to perform an update that will require manual updates (e.g. updating your path), and ask if they want to accept it.

[MikeMcQuaid]

Another thought (just trying to brainstorm ideas) would be to have a formula implement a variable that's last_compatible_version potentially and use that to compare against the version being upgraded from. Then we could do something like if installed_version < last_compatible_version we could then prompt the user with a dialogue or something that explains that you are about to perform an update that will require manual updates (e.g. updating your path), and ask if they want to accept it.

brew outdated does pretty similar already. We're never going to prompt with a dialogue.

[kkirsche]

Out of curiosity, why not? Yum and apt do it at times. It's not uncommon for package managers to do this type of thing

[MikeMcQuaid]

@kkirsche Homebrew never prompts users; it was an original design decision and at this point the consistency is internalised and not worth overriding for a single case.

[micahculpepper]

PSA: If you need to call python from inside an already-running python script, and, because of this change, #!/usr/bin/env python no longer works for you, you can find the executable that got you to where you are with sys.executable.

[kkirsche]

Fair enough @MikeMcQuaid — while it differs from my opinion on the situation, I respect the decision. Thanks for explaining that 👍

[genebean]

Just my two cents that it would have saved me a lot of time if this had been noted as a _breaking change_ in the output of brew upgrade or brew reinstall python or on the http://docs.brew.sh/Homebrew-and-Python.html. This bug report is the only reason I understand why my stuff just all of a sudden stopped working. I have no issue with the change, just the way in which it came about.

[acdha]

I wanted to third support for @DomT4's idea about grouping caveats at the end, at least for upgrades. In our case, the report is “brew upgrade breaks python” because everything output was well into the scrollback buffer behind the output from the remaining packages and if you're configuring virtualenvwrapper in your shell init files it's easy not to notice the change until you open a new shell window.

[ilovezfs]

This issue is about Python not where caveats go.

[kkirsche]

@MikeMcQuaid to address what seems to be a communication concern is that something that should get a new issue to attempt to track and hopefully address to reduce the impact from changes such as this?

[MikeMcQuaid]

@kkirsche No. If people want to submit a pull request to propose concrete changes we'll review it but we're not interested in any further post-mortem of this change.