Brew: PythonRequirement woes: Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x

[zmwangx]

See https://github.com/Homebrew/brew/pull/2917#issuecomment-317272358. I ran into this today myself, even with HOMEBREW_ENV_FILTERING turned on:

==> python -c import setuptools... --no-user-cfg install --prefix=/usr/local/Cellar/gnuradio/3.7.11_1/libexec/vendor --single-version-externally-managed --record=installed.txt
Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x!
     PYTHONPATH is currently: "/usr/local/Cellar/gnuradio/3.7.11_1/libexec/vendor/lib/python2.7/site-packages:/usr/local/lib/python3.6/site-packages"
     You should `unset PYTHONPATH` to fix this.

Reproducers

• site-packages dir for Python 3.x but you are running Python 2.x: https://github.com/Homebrew/brew/issues/2958#issuecomment-318661866
• site-packages dir for Python 2.x but you are running Python 3.x: https://github.com/Homebrew/brew/issues/2958#issuecomment-318917264

Diagnostics

I looked into python_requirement.rb and apparently it's picking up a wrong python_short_version when you have python3 as the default python on your PATH, which in turn inserts the wrong site-packages path into PYTHONPATH:

ENV["PYTHONPATH"] = "#{HOMEBREW_PREFIX}/lib/python#{short_version}/site-packages"

Note that python_short_version is decided from which_python, which in turn is basically which python_binary, where python_binary is "python"; and which is defined in requirement.rb:

  def which(cmd)
    super(cmd, PATH.new(ORIGINAL_PATHS))
  end

You immediately see what's wrong here: it's looking inside ORIGINAL_PATHS, and no wonder HOMEBREW_ENV_FILTERING doesn't help a bit.

Resolution

I'm on a tight deadline for something else so I stopped my investigation there — I didn't do a bisect, and nor did I really think the problem through. However, it seems that since we're doing

# Homebrew Python should take precedence over other Pythons in the PATH
ENV.prepend_path "PATH", Formula["python"].opt_bin
ENV.prepend_path "PATH", Formula["python"].opt_libexec/"bin"

we should really be putting those paths in front of ORIGINAL_PATHS when we do the which. This might be the source of inconsistency here.

In the meantime, my quick and dirty workaround is setting python_binary to "python2.7".

---

CC @DomT4 @ilovezfs

[zmwangx]

Oh I just realized I was running into a py3x site-packages dir for py2x problem, while Dom ran into the opposite py2x site-packages dir for py3x problem. That's kinda funny, I'm slightly confused as to how he managed to get a 2.7 short version when python_binary is explicitly "python3". Anyway, I don't have the time to investigate now.

[DomT4]

That's kinda funny, I'm slightly confused as to how he managed to get a 2.7 short version when python_binary is explicitly "python3"

The only change to my local setup around Python lately came post Mike's Python PR the other day when I chucked export PATH="$(brew --prefix python)/libexec/bin:$PATH" into my .zshrc. Otherwise I haven't changed anything else or set any new variables in the environment.

I tried to poke around various causes as to why it was happening to me but drew a consistent blank. If anyone has diagnostic ideas I'm happy to provide output against those where I can, but yeah, still a bit 🤷‍♂️ as to how I ended up where I did.

My default python is definitely Homebrew's python:

~> which python
/usr/local/opt/python/libexec/bin/python
~>  python --version
Python 2.7.13
~> python-config --prefix
/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7

[ilovezfs]

Have you checked whether a new user is affected?

[zmwangx]

@DomT4 Good old printf debugging may help:

diff --git a/Library/Homebrew/requirement.rb b/Library/Homebrew/requirement.rb
index cfb925b49a..bc76d2e194 100644
--- a/Library/Homebrew/requirement.rb
+++ b/Library/Homebrew/requirement.rb
@@ -151,6 +151,7 @@ class Requirement
   end

   def which(cmd)
+    puts "PATH for which: #{ORIGINAL_PATHS}"
     super(cmd, PATH.new(ORIGINAL_PATHS))
   end

diff --git a/Library/Homebrew/requirements/python_requirement.rb b/Library/Homebrew/requirements/python_requirement.rb
index c162b072c9..3cc8f367ab 100644
--- a/Library/Homebrew/requirements/python_requirement.rb
+++ b/Library/Homebrew/requirements/python_requirement.rb
@@ -15,6 +15,7 @@ class PythonRequirement < Requirement

   env do
     short_version = python_short_version
+    puts "python_short_version: #{python_short_version}"

     if !system_python? && short_version == Version.create("2.7")
       ENV.prepend_path "PATH", which_python.dirname
@@ -32,7 +33,9 @@ class PythonRequirement < Requirement
   end

   def which_python
+    puts "python_binary: #{python_binary}"
     python = which python_binary
+    puts "python: #{python}"
     return unless python
     Pathname.new Utils.popen_read(python, "-c", "import sys; print(sys.executable)").strip
   end

[MikeMcQuaid]

@zmwangx @DomT4 Can you both output what your PATH and (if set) PYTHONPATH are outside of Homebrew, too, and whether you have the Homebrew python and/or python3 installed? Thanks!

[zmwangx]

For me, I have ~/.pyenv/versions/3.6.1/bin in my PATH, which provides the first python. I also have both brewed python and python3. (Update: I don't typically have PYTHONPATH set when I run brew.) The diagnostics in my original post should explain this scenario fairly clearly; the question is how to fix it.

I'm confused about Dom's scenario though, and printing out some intermediate variables should shed some light.

[MikeMcQuaid]

@zmwangx And presumably you do not have a PYTHONPATH set manually, yes?

The diagnostics in my original post should explain this scenario fairly clearly; the question is how to fix it.

I don't have a series of reproducible steps yet, I'm trying to figure them out.

[zmwangx]

And presumably you do not have a PYTHONPATH set manually, yes?

No, I don't.

I'd imagine a reproducer could simply be:

brew install python python3
mkdir -p /tmp/debug-brew-python
ln -s /usr/local/bin/python3 /tmp/debug-brew-python/python
PATH=/tmp/debug-brew-python:$PATH
unset PYTHONPATH  # probably shouldn't matter; didn't check
brew install -s ...  # some formula that depends_on => :python and has python resources, e.g., ipython@5

Let me know if that doesn't cut it.

[ilovezfs]

@zmwangx so those directions do trigger the failure with watson although I still cannot reproduce the issue Dom was reporting in https://github.com/Homebrew/homebrew-core/pull/15941.

[zmwangx]

@ilovezfs Cool. As I said I'm confused about Dom's reverse issue since :python3 explicitly looks for python3. The intermediate variables should most likely reveal what's going on.

[MikeMcQuaid]

ln -s /usr/local/bin/python3 /tmp/debug-brew-python/python

🆒 so I guess that's the main reproducer: python is a python3, thanks!

[zmwangx]

Yep that's the point.

[DomT4]

Apologies for the delay here, and indeed a couple other Issues/PRs. Have been having some internet troubles this week which my ISP insist they've fixed 🙄.

I'm confused about Dom's scenario though, and printing out some intermediate variables should shed some light.

~> brew reinstall -s -v diffoscope
python_binary: python3
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/bin/python3
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
==> Reinstalling diffoscope
python_binary: python3
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/bin/python3
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
==> Using the sandbox
/usr/bin/sandbox-exec -f /tmp/homebrew20170728-14516-grnbas.sb nice /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/diffoscope.rb --verbose
python_binary: python3
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/bin/python3
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_short_version: 2.7
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python
python_binary: python
PATH for which: [#<Pathname:/Users/xyz/perl5/bin>, #<Pathname:/usr/local/opt/python/libexec/bin>, #<Pathname:/usr/local/opt/openssl/bin>, #<Pathname:/usr/local/opt/curl-max/bin>, #<Pathname:/Users/xyz/.opam/system/bin>, #<Pathname:/usr/local/var/rbenv/shims>, #<Pathname:/usr/local/bin>, #<Pathname:/usr/local/sbin>, #<Pathname:/usr/bin>, #<Pathname:/bin>, #<Pathname:/usr/sbin>, #<Pathname:/sbin>, #<Pathname:/usr/local/opt/go/libexec/bin>, #<Pathname:/Users/xyz/.cargo/bin>, #<Pathname:/Library/TeX/texbin>, #<Pathname:/Users/xyz/go/bin>]
python: /usr/local/opt/python/libexec/bin/python

Can you both output what your PATH and (if set) PYTHONPATH are outside of Homebrew, too, and whether you have the Homebrew python and/or python3 installed? Thanks!

/Users/xyz/perl5/bin:/usr/local/opt/python/libexec/bin:/usr/local/opt/openssl/bin:/usr/local/opt/curl-max/bin:/Users/xyz/.opam/system/bin:/usr/local/var/rbenv/shims:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/opt/go/libexec/bin:/Users/xyz/.cargo/bin:/Library/TeX/texbin:/Users/xyz/go/bin

And PYTHONPATH isn't set. Both Homebrew's python and python3 are installed.

[zmwangx]

@DomT4 I suppose you installed libmagic --with-python? I can reproduce the python_binary: python instances if that's the case, but not if libmagic is installed without python.

I didn't check the code, but it looks like when dependencies and requirements are collected recursively, the PythonRequirement of libmagic --with-python is being picked up, and its env is apparently overriding that of Python3Requirement from diffoscope.

[DomT4]

I suppose you installed libmagic --with-python?

👀 Yup. Built from source on 2017-05-24 at 14:56:51 with: --with-python.

[zmwangx]

To summarize, here's a repro for Dom's case:

brew install python python3
# Make sure the first python in PATH isn't python3, or the following step will fail.
brew reinstall libmagic --with-python
brew reinstall -s diffoscope

=>

...
Your PYTHONPATH points to a site-packages dir for Python 2.x but you are running Python 3.x!
...

[ilovezfs]

So the choices I see are

1) stop setting PYTHONPATH automatically in all cases
2) stop setting PYTHONPATH automatically if the dependency tree is mixed, or can be mixed, given some option combination
3) manually use ENV.delete("PYTHONPATH") in such cases
4) any other ideas you have

[DomT4]

3 seems like the least complicated one to implement, which limits potential for unintended consequences and/or major formulae rewriting.

[ilovezfs]

(3) seems riskless but it's a form of perpetual Whac-A-Mole without any evidence that it's necessary.

[zmwangx]

My original proposal still stands, I believe.

However, it seems that since we're doing

# Homebrew Python should take precedence over other Pythons in the PATH
ENV.prepend_path "PATH", Formula["python"].opt_bin
ENV.prepend_path "PATH", Formula["python"].opt_libexec/"bin"

we should really be putting those paths in front of ORIGINAL_PATHS when we do the which. This might be the source of inconsistency here.

I'll put together a patch tomorrow.

By the way, it won't fix https://github.com/Homebrew/brew/issues/2958#issuecomment-318917264, since that's rooted deeper in the dependency resolution chain.

[zmwangx]

I should add that none of the proposals above would fix https://github.com/Homebrew/brew/issues/2958#issuecomment-318917264 for the same reason.

[ilovezfs]

@zmwangx not sure what you're talking about. We know that ENV.delete("PYTHONPATH") works.

[zmwangx]

How does ENV.delete("PYTHONPATH") fix the problem where python2 is selected when python3 is expected?

[ilovezfs]

@zmwangx I'm talking about Dom's case.

[zmwangx]

Oh never mind, I was confused.

[ilovezfs]

Ah, OK.

Yeah, if you're talking about the python is python3 thing, your ORIGINAL_PATHS idea may work, or a rewrite of the python requirement that doesn't hard code the expectation of the binary name, or both.

[ilovezfs]

Another option would be to treat python like other versioned formulae, and disallow all mixed dependency trees, but that seems like an ideological approach.

[zmwangx]

hard code the expectation of the binary name

Not sure how that works.

disallow all mixed dependency trees

I would vote for that, but never put me in contact with all the people that would be frustrated by that change 😝

[ilovezfs]

Not sure how that works.

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/requirements/python_requirement.rb#L48-L50

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/requirements/python_requirement.rb#L62-L64

[zmwangx]

I meant what's the replacement for hard coded binary names?

[ilovezfs]

@zmwangx actually checking the version strings of each python*

[ilovezfs]

@zmwangx e.g., https://github.com/Homebrew/brew/blob/master/Library/Homebrew/gpg.rb#L3-L13

[zmwangx]

Sure. That's more reliable in a mixed dependency situation.

[ilovezfs]

@zmwangx yeah, I mean for solving your python == python3 thing.

[zmwangx]

Yeah, I was saying the ORIGINAL_PATHS fix should always work when Homebrew's python is installed, but could still break when there's no brewed python. (IMO It should still be implemented for consistency.) Checking each python* should work in all cases.

[MikeMcQuaid]

@zmwangx actually checking the version strings of each python*

This is what the RubyRequirement does. It would be good to make generic helpers in the base Requirement class to handle this stuff e.g. use which_all and then the @min_version, @max_version to keep comparing until you find one that matches.

[MikeMcQuaid]

stop setting PYTHONPATH automatically in all cases

This feels nicest to me if you can cope with the 🔥 😭 💥, @ilovezfs.

[ilovezfs]

This feels nicest to me if you can cope with the :fire: :sob: :boom:, @ilovezfs.

There's only one way to find out for sure.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[DomT4]

Leave it be bot. Still an issue AFAIK.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[MikeMcQuaid]

Closing this as it doesn't seem anyone is working on it.