Julia: failed to build julia against mkl on osx

[liugengdai]

Hi
I tried to build julia 0.4 on mac os with mkl. I made it successfully on Ubuntu but failed on osx.
The error is:
error during bootstrap:
LoadError(at "sysimg.jl" line 238: LoadError(at "util.jl" line 234: ErrorException("error compiling blas_vendor: could not load library "libmkl_rt"
dlopen(libmkl_rt.dylib, 1): image not found")))

It seems the mkl env is not set correctly. But I really follow the instruction on ReadMe.md. I think I've set MKLROOT properly.
Thank you and look forward to your help.

Gengdai

[ViralBShah]

I'm not sure this has been tried out much, or perhaps at all.

[albi3ro]

I've gotten it to work out, but had to figure out a couple things first. I've written it up here http://albi3ro.github.io/M4/programming/MKL.html

I'd like to know it that problem was specific to me, or helps your problem too. Seems like your problem is also in the environment variables though, so it might work.

[skleinbo]

I too get a similar error when trying to build against MKL

linalg/linalg.jl
LoadError("sysimg.jl",307,LoadError("linalg/linalg.jl",221,LoadError("linalg/blas.jl",79,ErrorException("error compiling vendor: could not load library \"libmkl_rt\"\ndlopen(libmkl_rt.dylib, 1): image not found"))))
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [/Volumes/scratch/juliamkl/usr/lib/julia/sys.o] Error 1

Make.user:

USE_INTEL_MKL = 1
USE_INTEL_MKL_FFT = 1

Environment variables

MKLROOT=/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl
LIBRARY_PATH=/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
DYLD_LIBRARY_PATH=/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/compiler/lib:/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib
CPATH=/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/include
NLSPATH=/opt/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib/locale/%l_%t/%N
TBBROOT=dfjndjnfd

TBBROOT is set to something not empty according to @albi3ro suggestion.
It's a fresh clone of the repo. Standard build against OpenBLAS works fine.

The following call fails in the function vendor() in base/linalg/blas.jl
cglobal((:MKL_Set_Num_Threads, Base.libblas_name), Void)

Funny enough, when called from a REPL (same shell, same environment variables), the library is loaded successfully

julia> cglobal((:MKL_Set_Num_Threads,"libmkl_rt"),Void)
Ptr{Void} @0x0000000313c4cdd0

Thanks for your time and help!
Stephan

[skleinbo]

Minimal example that reproduces the behavior:

$ julia -e "Libdl.dlopen(\"libmkl_rt\",1)"
$ DYLD_LIBRARY_PATH="" julia -e "Libdl.dlopen(\"libmkl_rt\",1)"
ERROR: could not load library "libmkl_rt"
dlopen(libmkl_rt.dylib, 5): image not found
 in dlopen at libdl.jl:36
 in process_options at /Volumes/scratch/Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib
 in _start at /Volumes/scratch/Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib

So I guess the variable is not passed down to the julia process that runs linalg.jl at the time of failure. Is this when the system image is build...?

I tried to comprehend the build process, but it's too convoluted for me to understand right now.

[tkelman]

One of its dependencies isn't being found without DYLD_LIBRARY_PATH set then?

[skleinbo]

Yes. Sounds rather trivial I guess. The point is that DYLD_LIBRARY_PATH _is_ set properly when the build is started.

At some point in the build process the value of the environment variable is forgotten / overwritten.
The process that eventually loads linalg/linalg.jl and fails is

juliamkl/usr/bin/julia -O3 -C native --output-o /Volumes/scratch/juliamkl/usr/lib/julia/sys.o --startup-file=no --sysimage /Volumes/scratch/juliamkl/usr/lib/julia/inference.ji sysimg.jl

(The repo is freshly cloned into /Volumes/scratch/juliamkl)

Thanks!

[Godisemo]

I have the exactly same issue as @liugengdai when I try to build against MKL. The way I found to solve it is to symlink MKL into the local julia lib folder:
ln -s /opt/intel/mkl/lib/libmkl_rt.dylib usr/lib/libmkl_rt.dylib
I don't know if this is a good idea or not, but it seems to work.

It seems as if the process of making the sysimg is not looking for libraries in the DYLD_LIBRARY_PATH at all. Should it?

[pablosanjose]

(Hi all, very first github post here!)

I also managed to build the latest master (0.6.0-dev.2867) under macOS 10.12.3 using the 2017 ifort compiler and MKL (not icc). It did require some tinkering. This is a summary of what I needed to do in addition to the README.md instructions :

1- Create a Make.user file with:

USEICC = 0
USEIFC = 1
USE_INTEL_MKL = 1
USE_INTEL_MKL_FFT = 1
USE_INTEL_LIBM = 1

2- Symbolic link to three dylib files that couldn't be found, despite the environment variables being correctly set

ln -s /opt/intel/mkl/lib/libmkl_rt.dylib usr/lib/libmkl_rt.dylib
ln -s /opt/intel/lib/libimf.dylib usr/lib/libimf.dylib
ln -s /opt/intel/lib/libintlc.dylib /usr/lib/libintlc.dylib

3- Invoke make with the following flag to solve an ifort error 10005 when building arpack ("unable to open argument file 'loader_path/'")

make LDFLAGS= -Wl

The versioninfo() output of the resulting build reads

Julia Version 0.6.0-dev.2867
Commit 9706c8b (2017-02-20 05:27 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin16.4.0)
  CPU: Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
  WORD_SIZE: 64
  BLAS: libmkl_rt
  LAPACK: libmkl_rt
  LIBM: libimf
  LLVM: libLLVM-3.9.1 (ORCJIT, ivybridge)

A quick test (computing one eigenvalue of a 10^5×10^5 SparseMatrixCSC) seems to show an approximate 15-20% advantage of this MKL build over the julialang.org libopenblas binary.

EDIT: not sure at all about that 20% advantage anymore. Other tests do not confirm this. See also https://discourse.julialang.org/t/benchmark-matlab-julia-for-matrix-operations/2000/92 for a more systematic discussion

[ViralBShah]

This works reasonably well now. Closing since the issue is about an old version of Julia. In any case, a new issue should be opened if an issue with newer Julia versions.

[pablosanjose]

Fantastic! Just out of curiosity (I struggled with this for a while): where was the problem exactly?

[ViralBShah]

Usually, hard to say on such old issues what fixed it. Also note that openblas 0.3 when merged on master should fix several performance issues on mac.

[pablosanjose]

I'm afraid I'm having the same issues as before. I don't see any difference with current master. Could you perhaps give us some pointers? The instructions for MKL compilation in https://github.com/JuliaLang/julia don't really work on MacOS with the free MKL libraries (unless I'm doing something dumb...)

[ViralBShah]

This is on master?

[ViralBShah]

It seems that you are not setting some env variable correctly.

[pablosanjose]

Yes, on today's master. Regarding env variables, mmm, perhaps, but I've tried all I could think of. I would expect that doing source /opt/intel/bin/compilervars.sh intel64 -platform mac should be enough, though, right? The discourse post I linked above explained what I'm seeing in detail.

[ViralBShah]

Yes, that ought to be. @bmharsha Can you spot something here that may be missing?

Also, @pabloferz Do you have some other LD_LIBRARY_PATH or DYLD_LIBRARY_PATH?

[pablosanjose]

I have

$ echo $DYLD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2018.2.164/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.2.164/mac/mkl/lib

However LD_LIBRARY_PATH is empty for me, should it not be?

(It seems other folks are having the same issues, see https://discourse.julialang.org/t/issues-building-julia-0-7-with-intel-mkl-on-macos/10159/7)

[bmharsha]

I haven’t tried building MKL Julia binaries on master yet, this is the configuration (And steps) that used to work for me on v0.6 , I've slightly modified these commands for v0.7

git clone https://github.com/JuliaLang/julia.git julia
cd julia

Create a Make.user file with following contents

USE_INTEL_MKL = 1
USE_INTEL_MKL_FFT = 1
USE_GPL_LIBS=0

source /opt/intel/bin/compilervars.sh intel64 -platform "mac"
make spawn="source /opt/intel/bin/compilervars.sh intel64 && 0.7.0"
cd contrib/mac/app
make spawn="source /opt/intel/bin/compilervars.sh intel64 && 0.7.0" VERBOSE=1 TAGGED_RELEASE_BANNER="Julia Alpha" JULIA_ENABLE_DOCBUILD=0 JULIA_CPU_CORES=4 JULIA_TEST_MAXRSS_MB=600 MARCH=x86-64

[pablosanjose]

Thanks @bmharsha. Following your instructions I get an error from that 0.7.0 in spawn (/bin/sh: 0.7.0: command not found) just after attempting to link flisp

Removing it I get

$ make spawn="source /opt/intel/bin/compilervars.sh intel64"
    FLISP src/julia_flisp.boot
compilervars.sh [-arch] <arch> [-platform <platform>]

  <arch> must be one of the following:
      ia32           : Set up for IA-32 target.
      intel64        : Set up for Intel(R)64 target.
  <platform> must be of the following:
      linux          : Set to Linux* target.
      mac            : Set to macOS* target.(default)

If the arguments to the sourced script are ignored (consult docs
for your shell) the alternative way to specify target is environment
variables COMPILERVARS_ARCHITECTURE to pass <arch> to the script
and COMPILERVARS_PLATFORM to pass <platform>
make[1]: *** [julia_flisp.boot] Error 1
make: *** [julia_flisp.boot.inc.phony] Error 2

[ViralBShah]

Different versions of MKL??

[pablosanjose]

I did have other Intel products installed. I removed them all and installed only the latest MKL library. Same errors.

$ echo $MKLROOT
/opt/intel/compilers_and_libraries_2018.2.164/mac/mkl
$ echo $DYLD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2018.2.164/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.2.164/mac/mkl/lib

[MasonProtter]

I can confirm I have the same issue as Pablo after trying Harsha's advice.

[MasonProtter]

Is the 0.7.0 not in reference to the Julia version? I'd guess that 0.7.0 is not the appropriate identifier for the master. IS there a make command one could do instead?

[dlfivefifty]

I tried building 0.7-alpha with USE_INTEL_MKL = 1 and get the following error:

…
In file included from /Users/solver/Projects/julia7-mkl/src/cgmemmgr.cpp:7:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/ExecutionEngine/SectionMemoryManager.h:20:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/ExecutionEngine/RTDyldMemoryManager.h:18:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/ExecutionEngine/JITSymbol.h:24:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/Support/Error.h:21:
/Users/solver/Projects/julia7-mkl/usr/include/llvm/Config/abi-breaking.h:23:6: warning: 
      'LLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING' is not defined, evaluates to
      0 [-Wundef]
#if !LLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING
     ^
1 warning generated.
    CC src/llvm-api.o
In file included from /Users/solver/Projects/julia7-mkl/src/llvm-api.cpp:15:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/Analysis/TargetLibraryInfo.h:13:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/ADT/DenseMap.h:18:
In file included from /Users/solver/Projects/julia7-mkl/usr/include/llvm/ADT/EpochTracker.h:19:
/Users/solver/Projects/julia7-mkl/usr/include/llvm/Config/abi-breaking.h:23:6: warning: 
      'LLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING' is not defined, evaluates to
      0 [-Wundef]
#if !LLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING
     ^
1 warning generated.
    LINK usr/lib/libjulia.0.7.0.dylib
    CC ui/repl.o
    LINK usr/bin/julia
Creating usr/etc/julia/startup.jl
Copying in usr/share/man/man1/julia.1
/Users/solver/Projects/julia7-mkl/contrib/install.sh 755 /Users/solver/Projects/julia7-mkl/contrib/julia-config.jl /Users/solver/Projects/julia7-mkl/usr/share/julia/
    PERL base/pcre_h.jl
    PERL base/errno_h.jl
    PERL base/build_h.jl.phony
    PERL base/file_constants.jl
    PERL base/uv_constants.jl
    PERL base/version_git.jl.phony
make[1]: *** [/Users/solver/Projects/julia7-mkl/usr/lib/julia/libmkl_rt.dylib] Error 1
make: *** [julia-base] Error 2

Here are my environment variables:
```julia
[solver-mbook:~/Projects/julia7-mkl] solver% echo $DYLD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2018.3.185/mac/tbb/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/compiler/lib:/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib
[solver-mbook:~/Projects/julia7-mkl] solver% echo $MKLROOT
/opt/intel/compilers_and_libraries_2018.3.185/mac/mkl
````

[pablosanjose]

This is the same problem I encountered, yes. It seems like an issue with the path. The spawn option to make didn't help me. However, putting all the MKL dylibs in usr/lib/julia got passed this particular problem, as explained in the discourse post mentioned above. At the end, however, I cannot make it link the libmkl_intel_thread.dylib library, whatever I tried.

[pablosanjose]

I finally managed to build current master with MKL on macOS! Once I got to the make[1]: *** [/pathtojulialib/libmkl_rt.dylib] Error 1 I just did ln -sf /pathtomkllibs/libmkl_rt.dylib /pathtojulialib/libmkl_rt.dylib and compilation went through. I think the problem is clearly around line 158 of the Makefile, where that very ln -sf command is constructed, but for some reason doesn't go through.

The new versioninfo() doesn't show me the BLAS/LAPACK version now. How can I check that I'm indeed running the MKL runtime?

julia> versioninfo()
Julia Version 0.7.0-alpha.0
Commit 22590d529d* (2018-05-31 00:07 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.5.0)
  CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

EDIT: for completeness: in my case pathtomkllibs = /opt/intel/compilers_and_libraries_2018.2.164/mac/mkl/lib and pathtojulialib = /Users/pablo/build/juliamkl/usr/lib/julia

EDIT2: I just learnt about BLAS.vendor() and Base.liblapack_name. They both correctly report MKL now.

[ViralBShah]

Which Makefile are you referring to?

[pablosanjose]

The one in /base : https://github.com/JuliaLang/julia/blob/2a45839ef9c9bb393fff0824dd11c3bd542bb537/base/Makefile#L158

[EthanAnderes]

I can verify this working for me. The thing that ended up fixing it for me was replacing usr/lib/libmkl_rt.dylib (which worked on v0.6) with usr/lib/julia/libmkl_rt.dylib. The former wouldn't work for v0.7 but did for v0.6.

julia> versioninfo()
Julia Version 0.7.0-alpha.59
Commit 2a45839ef9* (2018-06-09 17:04 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.6.0)
  CPU: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)
Environment:
  JULIA_NUM_THREADS = 2
  JULIA_FFTW_PROVIDER = MKL

Just for reference here is what worked for v0.6

ln -sf /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib  usr/lib/libmkl_rt.dylib

ln -sf /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_intel_thread.dylib  usr/lib/libmkl_intel_thread.dylib

... and here is what works on v0.7

ln -sf /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_rt.dylib  usr/lib/julia/libmkl_rt.dylib

ln -sf /opt/intel/compilers_and_libraries_2018.3.185/mac/mkl/lib/libmkl_intel_thread.dylib  usr/lib/julia/libmkl_intel_thread.dylib

[MasonProtter]

Finally gave this a whirl, and building with MKL worked for me with @pablosanjose 's fix.

[dlfivefifty]

The fix worked for me too 🎉 Could someone make a PR to fix the make file?

Unfortunately, I don't know anything about Make files, otherwise I'd offer to do this.

[pablosanjose]

I've been poking around some more, and I think the problem is in line 153 of base/Makefile:

REALPATH=`$$(call spawn,$$(build_depsbindir)/libwhich) -p ./$$(notdir $$@)` && \

This resolves the real path of libmkl_rt.dylib by calling the shell command usr/tools/libwhich -p libmkl_rt.dylib. However, while that works in a normal shell, it fails silently inside the script. A similar problem was noted in #25400.

I've been unable to figure out why libwhich fails in this way, so pinging @vtjnash

EDIT: The error is failed to open library: dlopen(libmkl_rt.dylib, 1): image not found

[vtjnash]

libwhich is expecting to find the library in the build-time LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH) environment variable, and then makes a symlink to the result, so that setting LD_LIBRARY_PATH isn't necessary at run-time. Some of the code for MKL in the Makefile uses an older setup where we tried to embed the full (absolute) path to the .so inside the system image, which might be conflicting with this. Note however that on macOS these environment variables can no longer be passed to make—due to SIP—so doing this configuration is more difficult now.

[pablosanjose]

Does this mean that the solution to this issue involves changes beyond the makefile?
(I ask because I've been trying to tinker with the makefile (somewhat blindly) in the hope that the library path gets resolved)

[MasonProtter]

Has anyone succeeded in building Julia with MKL on v1.0.2? The steps here seem to not work for me.

[dlfivefifty]

Yes, in OS X.

[MasonProtter]

Yeah, I just realized I have a different problem as its failing to build for me even without MKL.

[dlfivefifty]

Are you on OS X Mojave? I had to wipe my brew installation (rm /usr/local/*) and start from scratch because gcc was broken. Also, restore the headers in /usr/include:

installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

[MasonProtter]

Yeah, I think that's the problem for me as well since this worked before I installed Mojave. I tried following your instructions but it didn't seem to help. Did you do anything other than nuking /usr/local/* and installing those headers?

[dlfivefifty]

I reinstalled brew after nuking /usr/local/*, including

brew install gcc
brew install cmake

Also xcode-select --install.

Also: make sure you are building in a fresh folder.

[MasonProtter]

Is the target of the instsller command suppoed to be /?

[dlfivefifty]

Maybe double check the “restore headers” documentation

[dlfivefifty]

Probably safest to just double click /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

[ViralBShah]

We should probably have a new issue for this if necessary.