Faiss: Make fails with errors on ubuntu 14.04 with OpenBlas

[AKSHAYUBHAT]

After installing openblas
apt-get update && apt-get install -y libopenblas-dev
and changing Makefile.inc
Here is my Makefile.inc
Makefile.inc : https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/blob/master/faiss/makefile.inc

make fails with following error

++ -fPIC -m64 -Wall -g -O3  -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c AutoTune.cpp -o AutoTune.o  
g++ -fPIC -m64 -Wall -g -O3  -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c AuxIndexStructures.cpp -o AuxIndexStructures.o  
ar r libfaiss.a hamming.o utils.o IndexFlat.o IndexIVF.o IndexLSH.o IndexPQ.o IndexIVFPQ.o Clustering.o Heap.o VectorTransform.o index_io.o PolysemousTraining.o MetaIndexes.o Index.o ProductQuantizer.o AutoTune.o AuxIndexStructures.o
ar: creating libfaiss.a
g++ -o tests/demo_ivfpq_indexing -fPIC -m64 -Wall -g -O3  -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp tests/demo_ivfpq_indexing.cpp libfaiss.a -g -fPIC  -fopenmp /usr/lib/libopenblas.so.0
libfaiss.a(utils.o): In function `faiss::matrix_qr(int, int, float*)':
/root/DVA/faiss/utils.cpp:1215: undefined reference to `sgeqrf_'
/root/DVA/faiss/utils.cpp:1220: undefined reference to `sgeqrf_'
/root/DVA/faiss/utils.cpp:1223: undefined reference to `sorgqr_'
libfaiss.a(VectorTransform.o): In function `faiss::OPQMatrix::train(long, float const*)':
/root/DVA/faiss/VectorTransform.cpp:603: undefined reference to `sgesvd_'
/root/DVA/faiss/VectorTransform.cpp:594: undefined reference to `sgesvd_'
libfaiss.a(VectorTransform.o): In function `faiss::PCAMatrix::train(long, float const*)':
/root/DVA/faiss/VectorTransform.cpp:284: undefined reference to `ssyev_'
/root/DVA/faiss/VectorTransform.cpp:289: undefined reference to `ssyev_'
collect2: error: ld returned 1 exit status
make: *** [tests/demo_ivfpq_indexing] Error 1

[mdouze]

Hi

OK, I repro the error in a docker Ubuntu 14.04.

Checking how to solve...

[mdouze]

On ubuntu 14, Lapack is not included in the openblas library. You need to install it separately and link with it:

apt-get install liblapack3

Then add to makefile.inc

BLASLDFLAGS=/usr/lib/libopenblas.so.0 /usr/lib/lapack/liblapack.so.3.0

I updated the instructions in makefile.inc.Linux

[AKSHAYUBHAT]

I could make cpu interfaces, but when I tried doing this on GPU, essentially the same OS (tensorflow docker container) I ended up with following error (both for make as well as make py, I could fix it so only showing make py one for brevity).

 ---> Running in 978634ac89bc
/usr/local/cuda-8.0//bin/nvcc -I..  -I /usr/local/cuda-8.0//targets/x86_64-linux/include/ -Xcompiler -fPIC -Xcudafe --diag_suppress=unrecognized_attribute -gencode arch=compute_35,code="compute_35" -gencode arch=compute_52,code="compute_52" -gencode arch=compute_60,code="compute_60" --std c++11 -lineinfo -ccbin g++ -DFAISS_USE_FLOAT16 -I/usr/include/python2.7/ -I/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ -shared -o ../python/_swigfaiss_gpu.so ../python/swigfaiss_gpu_wrap.cxx libgpufaiss.a ../libfaiss.a \
        -Xcompiler -fopenmp -lcublas \
        -Xlinker  /usr/lib/libopenblas.so.0 /usr/lib/lapack/liblapack.so.3.0
nvcc fatal   : Don't know what to do with '/usr/lib/lapack/liblapack.so.3.0'
make: *** [../python/_swigfaiss_gpu.so] Error 1
The command '/bin/sh -c make py' returned a non-zero code: 2

This happens due to -Xlinker being applied only to the first argument and not the second one. I edited makefile.inc to manually apply the flag to both libraries and it seems to have worked

https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/pull/8/files#diff-fdd137c6f1e943e56710eb16ee5701a7

# BLAS LD flags for nvcc (used to generate an executable)
BLASLDFLAGSNVCC=-Xlinker $(BLASLDFLAGS)

# Same, but to generate a .so
BLASLDFLAGSSONVCC=-Xlinker  $(BLASLDFLAGS)

By changing this into following (essentially repeating -Xlinker), the error went away.

# BLAS LD flags for nvcc (used to generate an executable)
BLASLDFLAGSNVCC=-Xlinker /usr/lib/libopenblas.so.0 -Xlinker /usr/lib/lapack/liblapack.so.3.0

# Same, but to generate a .so
BLASLDFLAGSSONVCC=-Xlinker  /usr/lib/libopenblas.so.0 -Xlinker /usr/lib/lapack/liblapack.so.3.0

[empty16]

Thanks @AKSHAYUBHAT, faiss works now!

[oldtree]

Thanks @AKSHAYUBHAT