Homebrew-core: Option to install with openblas removed from numpy and scipy

[glarchev]

• [x] Confirmed this is a problem with brew installing one, specific formula (not cask) and not every time you run brew? If it's a general brew problem please file this issue at https://github.com/Homebrew/brew/issues/new. If it's a brew cask problem please file this issue at https://github.com/Homebrew/caskroom/homebrew-cask/new
• [x] Ran brew update and retried your prior step?
• [x] Ran brew doctor, fixed all issues and retried your prior step?
• [x] Ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• [x] If brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

Hello,
It looks like it is no longer possible to install numpy and scipy with openblas. I believe that functionality got removed when numpy and scipy got moved to homebrew-core.

Previously, it was possible to do that via the following option:
brew install numpy --with-openblas

However, now the following warning appears:
Warning: numpy: this formula has no --with-openblas option so it will be ignored!

Please consider adding this option back in, it was really useful.

Thanks

Gist-logs:
https://gist.github.com/7d1f03b47f173a0af1b5b8b348eccbe5

[glarchev]

Just wanted to add that this install command does install numpy, but the installed version doesn't use OpenBLAS (and I would like it to).

[damonmaria]

Ended up here looking into getting Numpy to use LAPACK from homebrew rather than the system one which results in the following warning from scipy:
RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.

Seven years old. Really?

Since homebrew core has newer versions of both openblas and lapack, and the system provided versions appear to be ancient, why doesn't numpy by default depend on homebrew's openblas and lapack and then link to those?

[MikeMcQuaid]

it was really useful.

Can you elaborate on that, please? We want to provide as few options as possible because we can't test them with CI or provide binary packages for them so they increase our support load.

[damonmaria]

Couldn't both the goals of using Homebrew's Openblas (and Lapack) with numpy, and removing the --with-openblas option (and therefore using bottled build) be solved by numpy depending on Openblas (and Lapack) and using them? Shouldn't cause any issues as both Openblas and Lapack are key-only. Would also mean warnings (like in the comment above) wouldn't be thrown by scipy.

But quite possible I'm missing something as my knowledge in this area is pretty thin. Would that mean new Openblas or Lapack versions would force a re-build / re-bottling of numpy?

[glarchev]

Can you elaborate on that, please?

Sure. When numpy/scipy is installed on a Mac OS, by default it links against Apple's BLAS libraries (which are a part of Apple's Accelerate framework). This happens even if other BLAS implementations (such as OpenBLAS or ATLAS) are installed on the system.

The Accelerate implementation is inferior to OpenBLAS/ATLAS, because it's quite a bit slower -- my understanding is that for some operations the difference can be an order of magnitude or more. Many computational python packages that use numpy/scipy strongly recommend linking numpy/scipy to OpenBLAS or ATLAS -- otherwise performance is prohibitively slow, to the point of being unusable.

It is already possible to install OpenBLAS on Mac OS using Brew, and linking numpy/scipy to OpenBLAS installation used to be very easy with the --with-openblas option, until it disappeared. In fact, I'm not aware of any drawbacks that OpenBLAS has compared to the Accelerate implementation, so, if OpenBLAS is already installed on the system, it might make sense to have numpy/scipy link to it by default.

It is possible to get around the issue by building numpy/scipy from source, and manually instructing it to link to a specific BLAS -- however, this defeats the purpose of Brew.

[ilovezfs]

I think what's needed is a third-party OpenBLAS tap that has copies of all formulae from homebrew/core and homebrew/science that can use OpenBLAS on macOS.

See the documentation for how to create and maintain a tap.

[MikeMcQuaid]

@glarchev Thanks for the explanation! I agree with @ilovezfs here.

[damonmaria]

Hi @ilovezfs and @MikeMcQuaid. I would be willing to contribute to something like the tap you describe. I've had a quick read of the docs regarding writing formula and contributing.

Would all the formulae in this new tap have to be marked as conflicting with the core / science versions? Or, if the new tap versions were keg-only would that get around the problem?

Would it be asking for trouble if a user installed a package that depended on the system BLAS, and another that depended on homebrew OpenBLAS, and then tried to use the 2 packages together? Say 2 Python formula/packages that both linked to BLAS used in the same Python project?

Also, I understand there's probably a reason for this so forgive my noob understanding but it would help with developing this new tap to understand the reasoning. Given formula like numpy link to BLAS no matter what, of which there's a system version that is well out of date, and a homebrew version that is current, why doesn't the numpy formula require the OpenBLAS formula and link to that? I gather OpenSSL is an example where it is always done this way (all homebrew formula relying on OpenSSL link to the homebrew version) but security concerns are obviously heightened there. Why are other 'base' dependencies like BLAS not handled the same way?

[MikeMcQuaid]

Why are other 'base' dependencies like BLAS not handled the same way?

We prefer to use the system versions when possible to reduce install time and the number of dependencies.

[adamberenzweig]

@damonmaria did you make any progress with a numpy tap that's built against openBLAS? We're hitting a bug in the Accelerate version of numpy, but we'd like to keep managing our python installations with brew if possible.

The bug is described more here, FYI: https://mail.python.org/pipermail/numpy-discussion/2017-July/077079.html

[damonmaria]

@adamberenzweig No. That slipped way too far down the priority list for us sorry.

[adamberenzweig]

Thanks for the reply. For anyone interested, conda-forge's numpy package is
built against openBLAS, and it solved the problem I was seeing.

On Mon, Jan 22, 2018 at 5:50 PM, Damon Maria [e-mail]
wrote:

@adamberenzweig https://github.com/adamberenzweig No. That slipped way
too far down the priority list for us sorry.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Homebrew/homebrew-core/issues/16253#issuecomment-359597986,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWAUSDruE7MCxgM3WJxgl_9zv89NSeRks5tNRDOgaJpZM4OoyTy
.