Staged-recipes: Intel released the development packages for MKL; is it possible to add MKL support for conda-forge packages?

[dmitry-kabanov]

Intel released the development packages for MKL with yum, apt, and conda:

https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/699815#comment-1905276
https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python

So my question is: is it possible to add support for MKL in the conda-forge packages?

[dirmeier]

+1

[patricksnape]

Very interesting - I guess we would need confirmation from Intel that they would be OK with us distributing their binaries on our channel. Looks like (from that linked comment that I missed) they claim the packages are distributed under a free and permissive license - though I'd like to see a copy of that License.

I would be very supportive of this as it would solve many issues on Windows.

[patricksnape]

@jakirkham Maybe this could be a solution to having BLAS on Windows?

[jakirkham]

though I'd like to see a copy of that License.

I agree with you here, @patricksnape. If the license requires us to indemnify Intel (has been the case with their licenses in the past), then that would be a problem. For instance, if one of our users sues them, then I think that is a non-starter (i.e. we have no funds to do this with).

[rscohn2]

Here are the licenses:
https://software.intel.com/en-us/articles/end-user-license-agreement

Intel is ok with you distributing the binaries. However, we already publish the binaries under the intel channel on anaconda so I think it would be better to just put a dependence on them.

[jakirkham]

Here are the licenses:
https://software.intel.com/en-us/articles/end-user-license-agreement

Will take some time to parse all of this.

Admittedly IANAL. However this text is generally what makes me and other nervous. I don't want to get conda-forge in hot water over how we build NumPy against MKL for instance if it should cause issues on some end user's computer and they choose to sue Intel. IOW we don't want to end up having to pay for legal expenses for Intel in this case. Any thoughts on this point?

(v) will indemnify, hold harmless, and defend Intel and its suppliers from and against any claims
or lawsuits, including attorney's fees, that arise or result from Your modifications, derivative
works or Your distribution of Your Product.

However, we already publish the binaries under the intel channel on anaconda so I think it would be better to just put a dependence on them.

My thoughts on this are still incomplete, but some of them are below.

This is certainly an interesting idea and probably the best way forward. Though would be good to hear from others ( cc @conda-forge/core ). We would probably need to make some tweaks to get this to work right. The most notable is we would need to add an mkl feature to the blas ( cc @pelson ).

Also would have to think about this a bit as we can't use mkl everywhere. For instance we have GPL packages that rely on some form of BLAS, but it would not be ok for them to link to MKL. So we would want to make this clear to the user somehow should they try to change BLASes in such an environment. Also we would need to discuss how best to enact a policy to avoid any maintainer mistakes on this front. Maybe have the linter ping core when a package is used from the intel channel. IDK if we could require a PR review in just these cases.

Would also add that we would want to make the intel channel lower priority than conda-forge. Mentioning this as I see things like the notebook package in the intel channel, which we would be wanting to get from conda-forge. Also we wouldn't want some package that the intel chooses to incorporate down the road taking priority over the equivalent conda-forge package. Though there is then the question of which gets higher priority defaults or intel? I'd lean toward making intel higher priority than defaults to make sure that we are getting intel's packages and not defaults copies of such packages.

Others should definitely weigh in on any/all of this.

[rscohn2]

Pointing out that the intel channel has a MKL package was probably a distraction. Anaconda also provides a MKL package so you can get it from defaults and there is nothing special about our version of MKL libraries. We are working with continuum to eliminate duplicated packages between the 2 channels when there is no difference.

Continuum uses the nomkl feature and publishes numpy builds with MKL and openblas so the end user has control. That seems like a good solution for conda-forge.

[jakirkham]

Pointing out that the intel channel has a MKL package was probably a distraction.

Hardly. This is an often requested feature. So it is pretty important that we do something on this front.

Anaconda also provides a MKL package so you can get it from defaults and there is nothing special about our version of MKL libraries.

Right, but defaults doesn't supply the headers IIRC. So we won't be able to compile new things against that package. Though this could have changed recently.

We are working with continuum to eliminate duplicated packages between the 2 channels when there is no difference.

In which direction?

Continuum uses the nomkl feature and publishes numpy builds with MKL and openblas so the end user has control. That seems like a good solution for conda-forge.

Unfortunately this doesn't work well when there are multiple options. Hence why we have gone with the blas metapackage that we have now. So integration of mkl into conda-forge as of right now would be adding mkl as one of our blas features.

[rscohn2]

The direction is to have everything available in defaults, including mkl headers. We would continue to publish numpy and other packages where we have changes that are not yet available upstream.

So integration of mkl into conda-forge as of right now would be adding mkl as one of our blas features.

If there is something Intel can do to make this happen, let me know.

[jakirkham]

So this doesn't pertain to mkl per se, but I noticed that openmp is in the intel channel. Is this LLVM's OpenMP or is this Intel's? I'm guessing the latter. Part of the reason I ask is we are about to add LLVM's OpenMP to conda-forge. It would be good if we could disambiguate these sorts of things. Having packages named openmp will lead to confusion as LLVM's OpenMP seems to already confusingly be called openmp in a few cases (e.g. Homebrew).

[rscohn2]

It's Intel openmp. We will consider renaming it.

[tadeu]

Sorry for reviving this thread but I couldn't find a better place for this discussion: what is the current status on adding MKL support?

I'm specially interested in MKL support in order to have a BLAS/LAPACK on Windows that SciPy is able to use. Currently, OpenBLAS LAPACK seems broken in a few places (when used by SciPy) [see https://github.com/conda-forge/scipy-feedstock/pull/78]. I know a lot of effort is going into it, and it is super nice to have an open-source BLAS and LAPACK implementation that is competitive with MKL, but OpenBLAS currently seems to be extra-hard to maintain on Windows (always a lot of patches needed, and hacks with mingw32 or flang).

I've noticed that the conda-forge channel already has some packages coming from Intel: https://anaconda.org/conda-forge/mkl_fft, https://anaconda.org/conda-forge/mkl_random. If mkl-static, mkl-devel packages (and dependencies) were added, I think this would enable to start building NumPy and SciPy versions with MKL. What do you think?

[tomashek]

Hello Tadue. Is it an option to use mkl-devel, mkl-include, and mkl-devel directly from the Intel channel on anaconda.org (https://anaconda.org/intel)? If not there, what if we worked with Anaconda to have them available on the default Anaconda repository? Would that provide the ease of access that you require?

In the meantime, we are at reaching out to the conda-forge folks to understand what the roadblocks are. Certainly use of the Intel channel is the easiest because it is available now, though I think it possible to move toward other options relatively quickly.

Best,
Todd

[msarahan]

mkl-devel is already available on defaults:

conda search mkl-devel
Fetching package metadata ...........
mkl-devel                    2017.0.4             ha5f1040_0  defaults
                             2018.0.0             hd3f8bfc_4  defaults
                             2018.0.0             ha93b67f_4  defaults
                             2018.0.0             hcb2b0c4_6  defaults
                             2018.0.1             h5cfdff2_4  defaults
                             2018.0.1             h470d631_5  defaults
                             2018.0.2                      1  defaults

[tomashek]

Of course, thank you, Mike.

Tadeu, can you clarify where the gap is that you see? It is likely I do not understand your use case.

[djsutherland]

IIRC, there are two problems with using MKL on conda-forge right now:

• Add MKL as an option to the blas metapackage, or choose some other scheme based on build variants. Maybe add a little infrastructure around that.

• More importantly: figure out if the issue with the license (https://github.com/conda-forge/staged-recipes/issues/3022#issuecomment-312001293) is actually a problem. Though of course nothing is likely to happen, conda-forge can't be taking legal responsibility for things since we have no funding. If we just depend on the mkl package from defaults, does that actually "count" as us not distributing it, or is distributing things linked to mkl enough?

[jjhelmus]

I believe a the problem of a Fortran compiler still need to be solved in order to provide MKL builds of SciPy in Windows. IIRC, MKL accelerated NumPy packages can be build without a Fortran compiler on Windows.

[tadeu]

Although mkl-devel is on defaults, mkl-static is still not, and it seems that mkl-static is the package that contains BLAS and LAPACK which can be linked against.

Although we can mix channels when _using_ conda-forge, I think it is supposed to be self-contained when building packages, i.e., it can't build packages that depend on packages from other channels. Is this assumption correct? Perhaps it can depend on defaults?

[jakirkham]

@tadeu, @tomashek and I spoke about a bit about how we might make use of MKL in conda-forge packages about a month ago, which we agreed would be desirable. Sadly I've not had time to follow up. Though still would like to. Hoping to resume that conversation in a month when other near term demands have been met.

Would add that I think @dougalsutherland has done a nice job of summarizing the situation above.

If people have thoughts on what would be a better solution to the blas package, that would be a great community topic to dig into (possibly in a new issue on the webpage repo). 😉

As to the legal issues, is there someone at NumFOCUS we could talk to? I'm guessing no one here moonlights as a lawyer. 😄

The Fortran compiler is a special issue w.r.t. SciPy. Though wouldn't be surprised if that comes up again. There has been some effort to use a development version of Flang to address this need on Windows while keeping UCRT compatibility. Another option would be MinGW-w64's GFortran compiler. My understanding, which may very well be outdated, is this is not compatible with UCRT, which was a blocker. Would be worthwhile to reinvestigate whether this is still true of the MinGW-w64 toolchain and whether there are development versions that would work. If we could get a copy of Intel's compilers on CIs, that might be another way forward. Though this last option probably is a bit tricky and would require a fair bit of discussion. Have not looked into cross-compilation, but there might be some good Linux to Windows options that would be worth exploring as well.

Hopefully this status update/suggested paths forward helps provide context as to where this issue stands and what paths there are to progress on it.

[tomashek]

@tadeu , mkl-devel should have everything needed to build using dynamic linking (shared libs, headers, link-libs on Windows). mkl-static is only needed if you want to use static linking. If you find this to not be the case for you, I am happy to investigate.

[tadeu]

(Perhaps this is kind of off-topic now, but I don't know a better place to reply it, so here it goes):

@tadeu , mkl-devel should have everything needed to build using dynamic linking (shared libs, headers, link-libs on Windows). mkl-static is only needed if you want to use static linking. If you find this to not be the case for you, I am happy to investigate.

@tomashek, at least up to version 2018.0.2, considering the win64 package, the mkl package have a lot of libs to link against in Library/lib (such as scalapack and blacs), but it is missing lapack95 and blas95, which are present in their static versions in the mkl-static package (mkl_blas95_ilp64.lib, mkl_blas95_lp64.lib, mkl_lapack95_ilp64.lib, mkl_lapack95_lp64.lib).

[tadeu]

@jakirkham thanks for the long explanation! Looking forward for the follow up ;)

[tomashek]

@tadeu, you are correct that you will need the mkl-static package for those libraries on Windows. Would it help if the Anaconda folks added the mkl-static package to defaults? Or is that no different for your use case than it being available on the Intel channel?

[djsutherland]

@tomashek, it's useful for potential conda-forge usage for things to be on defaults: we (at least currently) do pull things in from defaults when they're not available in conda-forge, both on the CI build processes and by assumption on users' setups, but not sure adding intel as a fallback is a decision we want to make (particularly if it's only for mkl-static).

[anton-malakhov]

I'd advocate for dynamic linkage with mkl_rt library as well. It allows runtime decisions what threading layer to use, which in turn allows tbb4py to handle the oversubscription: https://github.com/conda-forge/tbb-feedstock/issues/13#issuecomment-394840309 Cc @oleksandr-pavlyk

[sveinugu]

+1 for mkl support in conda-forge. Any updates?

[scopatz]

According to our legal counsel via NumFOCUS, we are allowed to redistribute the MKL and are therefore able to distribute packages linked against it. At this point, we really need a champion to step up an focus on it :) @isuruf and I can try to help guide anyone who wants to help! I'd start by checking out the work @isuruf has been doing in https://github.com/conda-forge/blas-feedstock/pull/16

[isuruf]

This is possible now and many packages support switching blas implementations as in https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation

[jjhelmus]

This is great. Thanks for all you work on this @isuruf.

[tomashek]

Note that the Intel licenses have changed since this discussion was hashed out. The indemnification clause has changed from what is quoted above to:

YOU AGREE TO INDEMNIFY AND HOLD INTEL HARMLESS AGAINST ANY CLAIMS AND EXPENSES RESULTING FROM YOUR USE OR UNAUTHORIZED USE OF THE SOFTWARE.

Note that the phrase "and defend Intel and its suppliers from and against any claims" is no longer there. IANAL, but I do not see, in the current ISSL, any provision that would require conda-forge to incur legal expenses on Intel's behalf.

[tomashek]

@jakirkham @scopatz , in light of the current license and NumFOCUS counsel, are there remaining roadblocks to use of Intel(R) MKL and other similarly licensed Intel software? We are in the process of reviewing licensing and would welcome your feedback for consideration.

[scopatz]

You are free to use MKL within conda-forge packages. Some already do!