Brew: Virtualenv.do_install does not work for various Python packages

[adamtheturtle]

• [X] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
• [X] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Homebrew/homebrew-core: https://github.com/Homebrew/homebrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [X] ran brew update and can still reproduce the problem?
• [X] ran brew doctor, fixed all issues and can still reproduce the problem?
• [X] ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

I am a maintainer of an application which ships a Homebrew recipe so that users can easily install a CLI on macOS.
The application is written in Python 3.
The application has multiple dependencies.

The recipe has the following layout, but with many more resources:

minidcos.rb:

class Minidcos < Formula
  include Language::Python::Virtualenv

  url "https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1"
  homepage "http://minidcos.readthedocs.io/en/latest/"
  depends_on "python3"
  depends_on "pkg-config"

  resource "httplib2" do
    url "https://files.pythonhosted.org/packages/67/33/29779c5aaeac796679a37bf798b3c2adbfaae7dbf13e966b0ab9c3aa06c0/httplib2-0.12.3.tar.gz"
    sha256 "a18121c7c72a56689efbf1aef990139ad940fee1e64c6f2458831736cd593600"
  end

  def install
    virtualenv_install_with_resources
  end

end

Some of those resources specify a custom build-backend in their [build-system] section in their pyproject.toml.

There is a pip bug which prevents these packages from installing with --no-binary :all:.

Homebrew's Virtualenv.do_install uses pip's --no-binary :all: option:

https://github.com/Homebrew/brew/blob/6b2dbbc96f7d8aa12f9b8c9c60107c9cc58befc4/Library/Homebrew/language/python.rb#L260-L264

It is not clear to me why --no-binary :all: is used in Homebrew, so I don't want to just say "Homebrew would be better without this option". Perhaps this is a philosophical decision.

However, links on that pip issue show that folks are using workarounds to avoid using --no-binary all.

My hope is that either this option can be removed, or there can be an easy option for recipe maintainers to avoid --no-binary :all:.

What happened (include command output)

Command output

~/D/foobar ~> brew install minidcos.rb

  ==> Downloading https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1

  Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/4b8bccbb4ca6db124ea6a6abdfe0968853065133307c5fe8bce33d4112f9c168--dcos-dcos-e2e-2019.05.24.1-0-gfeb3f88.tar.gz

  Warning: Cannot verify integrity of 4b8bccbb4ca6db124ea6a6abdfe0968853065133307c5fe8bce33d4112f9c168--dcos-dcos-e2e-2019.05.24.1-0-gfeb3f88.tar.gz

  A checksum was not provided for this resource

  For your reference the SHA-256 is: f30be86f3b773d60ebe7bf8b772df31872238c24f4c84e8f5e8944810ccd0caf

  ==> Downloading https://files.pythonhosted.org/packages/53/c0/c7819f0bb2cf83e1b4b0d96c901b85191f598a7b534d297c2ef6dc80e2d3/virtualenv-16.6.0.tar.gz

  Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/6d18e6ad1a4c8e31dfc412cb5513a676119ead57a6ad5c830a0ec57ac08fd763--virtualenv-16.6.0.tar.gz

  ==> python3 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/minidcos--homebrew-virtualenv-20190601-55148-1189zyg/target --install-scripts=/private/tmp/minidcos--homebrew-virtualenv-20

  ==> python3 -s /private/tmp/minidcos--homebrew-virtualenv-20190601-55148-1189zyg/target/bin/virtualenv -p python3 /usr/local/Cellar/minidcos/2019.05.24.1/libexec

  ==> Downloading https://files.pythonhosted.org/packages/1f/87/9ea76ab4cdf1fd36710d9688ec36a0053067c47e753b32272f952ff206c5/flit-1.3.tar.gz

  Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/588d227f6a17845ac2092aedddac49e5349fb8390ee22121f57910130a52a475--flit-1.3.tar.gz

  ==> /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/pip install -v --no-deps --no-binary :all: --ignore-installed /private/tmp/minidcos--flit-20190601-55148-1g5um1n/flit-1.3

  Last 15 lines from /Users/adam/Library/Logs/Homebrew/minidcos/03.pip:

  Removed build tracker '/private/tmp/pip-req-tracker-laxe52t6'

  ERROR: Command "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/tmp/pip-req-build-id_d4tsl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-lf57luvq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/../include/site/python3.7/flit" failed with error code 1 in /private/tmp/pip-req-build-id_d4tsl/

  Exception information:

  Traceback (most recent call last):

    File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main

      status = self.run(options, args)

    File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 414, in run

      use_user_site=options.use_user_site,

    File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs

      **kwargs

    File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 951, in install

      spinner=spinner,

    File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess

      % (command_desc, proc.returncode, cwd))

  pip._internal.exceptions.InstallationError: Command "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/tmp/pip-req-build-id_d4tsl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-lf57luvq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/../include/site/python3.7/flit" failed with error code 1 in /private/tmp/pip-req-build-id_d4tsl/

Do not report this issue to Homebrew/brew or Homebrew/core!

What you expected to happen

No error.

Step-by-step reproduction instructions (by running brew commands)

Create the following file minidcos.rb.

class Minidcos < Formula
  include Language::Python::Virtualenv

  url "https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1"
  homepage "http://minidcos.readthedocs.io/en/latest/"
  depends_on "python3"
  depends_on "pkg-config"

  resource "flit" do
    url "https://files.pythonhosted.org/packages/1f/87/9ea76ab4cdf1fd36710d9688ec36a0053067c47e753b32272f952ff206c5/flit-1.3.tar.gz"
    sha256 "6f6f0fb83c51ffa3a150fa41b5ac118df9ea4a87c2c06dff4ebf9adbe7b52b36"
  end

  def install
    virtualenv_install_with_resources
  end

end

Run brew install minidcos.rb.

brew configuration

~ ~> brew config
HOMEBREW_VERSION: 2.1.1
ORIGIN: https://github.com/Homebrew/brew
HEAD: b4f73e61649fcfc5aaa779c311e2514619ce01e7
Last commit: 7 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 98167e409139cf6fed8c289600ba079473bba6b6
Core tap last commit: 5 weeks ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_LOGS: /Users/adam/Library/Logs/Homebrew
CPU: quad-core 64-bit skylake
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 10.0 build 1001
Git: 2.20.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
macOS: 10.14.5-x86_64
CLT: 10.2.1.0.1.1554506761
Xcode: 10.2.1
CLT headers: 10.2.1.0.1.1554506761
~ ~> brew doctor
Your system is ready to brew.

[MikeMcQuaid]

@adamtheturtle In future please amend the existing issue and comment rather than opening multiple issues, thanks.

It is not clear to me why --no-binary :all: is used in Homebrew, so I don't want to just say "Homebrew would be better without this option". Perhaps this is a philosophical decision.

You can check the commit logs, they may provide some enlightenment there.

My hope is that either this option can be removed, or there can be an easy option for recipe maintainers to avoid --no-binary :all:.

We are unlikely to remove this unless it's needed for Homebrew/homebrew-core formula. It may be a better option for your formula to do the same underlying steps done by virtualenv_install_with_resources but manually where you can pass whatever pip options you like. virtualenv_install_with_resources is not meant to handle all use cases.

Closing this but can continue discussion here, thanks!

[adamtheturtle]

@adamtheturtle In future please amend the existing issue and comment rather than opening multiple issues, thanks.

Will do.

You can check the commit logs, they may provide some enlightenment there.

I may be missing something but I believe the commit at https://github.com/Homebrew/brew/commit/2783adec4a906c8fb5c45aa1305b1460b5bc8a5b#diff-86d78c9fcb1fe847e0ef982ea0b5adfd added this option without description.

We are unlikely to remove this unless it's needed for Homebrew/homebrew-core formula. It may be a better option for your formula to do the same underlying steps done by virtualenv_install_with_resources but manually where you can pass whatever pip options you like. virtualenv_install_with_resources is not meant to handle all use cases.

That's fair, and your suggestion is what many folks are doing, as shown by the linked issues to the pip issue.

However, as a formula author this is a shame because it used to be so easy! Hopefully the pip bug will be fixed soon.

Thank you for your response @MikeMcQuaid !

[MikeMcQuaid]

I may be missing something but I believe the commit at 2783ade#diff-86d78c9fcb1fe847e0ef982ea0b5adfd added this option without description.

Yes. @tdsmith or @jonchang @lembacon any thoughts on this commit and issue?

[adamtheturtle]

For anyone following, I tried @MikeMcQuaid 's suggestion.
This did not work because I hit pip #6264 and saw AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'.

Instead, I replaced my formula's install section with the following which installs an old pip:

  def install
    wanted = %w[python python@2 python2 python3 python@3 pypy pypy3].select {{ |py| needs_python?(py) }}
    raise FormulaAmbiguousPythonError, self if wanted.size > 1

    python = wanted.first || "python2.7"
    python = "python3" if python == "python"
    venv = virtualenv_create(libexec, python.delete("@"))
    venv.instance_variable_get(:@formula).system venv.instance_variable_get(:@venv_root)/"bin/pip", "install",
                    "-v", "--no-deps",
                    "--ignore-installed",
                    "--upgrade",
                    "--force-reinstall",
                    "pip<19"
    venv.pip_install resources
    venv.pip_install_and_link buildpath
    venv
  end

[jonchang]

Passing --no-binary during pip install disables using compiled python packages, which I think is behavior we want to keep.

[tdsmith]

Yeah, avoiding precompiled binaries was the goal, in keeping with the build-from-source ethos. I'm not sure whether Homebrew can do something clever here. I'm moving today (on truck-watch duty atm!) -- I can try and take a look later this week, but if someone else has any interest in investigating, don't wait for me :)

[MikeMcQuaid]

Thanks all!

[adamtheturtle]

Thank you from me as well for spending the time on this. Perhaps I misunderstood - I am by no means a packaging expert - but I understood from Building from Source in Homebrew and Chocolatey that build from source is not necessarily the direction that Homebrew is going in. (No need to spend time correcting me as I haven't done thorough research on this).

[MikeMcQuaid]

We build our own packages from source but distribute them as binaries. This is differentiated from distributing upstream binaries (which is done by brew cask).

[jfoy]

More context on where --no-binary can cause problems with build-time dependencies: https://github.com/pypa/pip/issues/5739