Faiss: How It is Possible to Import and Use Faiss in Google Colab?

[AhmadM-DL]

How one can import and use Faiss in Google's Colab.

I tried the following:

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local
!conda install faiss-cpu -c pytorch

But I it is not working.

Any thoughts?

[beauby]

@AhmadM-DL Could you post the output of the above commands in a gist?

[AhmadM-DL]

Sure,

Here is a link to the gist:
https://gist.github.com/AhmadM-DL/c0ad39dd03f3970f594fd0fe112e7932

And here is a link to the notebook:
https://colab.research.google.com/drive/1nXTPpZ5NWuuEvXpp3JfXcVgP4gIPbvbn

FAISS is being imported. However, after running the following code snippet:

D, I = index.search(xq, k)     # actual search
print(I[:5])                   # neighbors of the 5 first queries
print(I[-5:])                  # neighbors of the 5 last queries

The colab session crashes with run time logs:

Jul 15, 2019, 7:11:27 PM | WARNING | WARNING:root:kernel e85c6d26-c332-4074-8459-5633f7be8017 restarted
-- | -- | --
Jul 15, 2019, 7:11:27 PM | INFO | KernelRestarter: restarting kernel (1/5), keep random ports
Jul 15, 2019, 7:11:25 PM | WARNING | Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Jul 15, 2019, 7:02:42 PM | WARNING | WARNING:root:kernel e85c6d26-c332-4074-8459-5633f7be8017 restarted
Jul 15, 2019, 7:02:42 PM | INFO | KernelRestarter: restarting kernel (1/5), keep random ports
Jul 15, 2019, 7:02:42 PM | WARNING | Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Jul 15, 2019, 6:56:41 PM | INFO | Adapting to protocol v5.1 for kernel e85c6d26-c332-4074-8459-5633f7be8017
Jul 15, 2019, 6:56:39 PM | INFO | Kernel started: e85c6d26-c332-4074-8459-5633f7be8017
Jul 15, 2019, 6:56:16 PM | INFO | Kernel interrupted: eb8c1782-3635-4a47-b4d9-167752512e75
Jul 15, 2019, 6:55:51 PM | INFO | Adapting to protocol v5.1 for kernel 24a187fe-1712-4616-863e-e38b2bdd9339

The problem resides in MKL's error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

[AhmadM-DL]

Note that executing using command line script does execute properly.

!python -c "

import faiss;
import numpy as np;
x = np.random.random((1000, 321)).astype('float32');
ncentroids = 10; 
niter = 20; 
verbose = True; 
d = x.shape[1]; 
kmeans = faiss.Kmeans(d, ncentroids, niter=niter, verbose=verbose); kmeans.train(x);

"

It outputs:

Clustering 1000 points in 321D to 10 clusters, redo 1 times, 20 iterations
Preprocessing in 0.00 s
Iteration 19 (0.01 s, search 0.01 s): objective=26131.1 imbalance=1.476 nsplit=0  

[mdouze]

We are working on a workaround.

[AhmadM-DL]

Is there any new updates related to this issue?

[lobpcg]

!apt install libomp-dev
!python -m pip install --upgrade faiss faiss-gpu
import faiss

seems to work just fine in Colab...

[ultrasounder]

!apt install libomp-dev
!python -m pip install --upgrade faiss faiss-gpu
import faiss

seems to work just fine in Colab...

Can confirm this solution works as of April2020.Thanks

[lobpcg]

I do not have a Colab account any more.

[ultrasounder]

Sorry, I meant to say "I can confirm that the solution proposed by You still works though it was back in March". Thanks

[korakot]

I tested and simplified it a bit.

For Colab CPU Instance:

!apt install libomp-dev
!pip install faiss

For Colab GPU-Instance:

!pip install faiss-gpu

[enaserianhanzaei]

None of these solutions worked for me. So what I did, was to install the conda and then install the faiss via it, which is a painful procedure.

1:
%env PYTHONPATH=

2:
%%bash
MINICONDA_INSTALLER_SCRIPT=Miniconda3-4.5.4-Linux-x86_64.sh
MINICONDA_PREFIX=/usr/local
wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT
chmod +x $MINICONDA_INSTALLER_SCRIPT
./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX

3: modify it based on your python version
%%bash
conda install --channel defaults conda python=3.6 --yes
conda update --channel defaults --all --yes

4:
import sys
_ = (sys.path.append("/usr/local/lib/python3.6/site-packages"))

5:
!conda install faiss-cpu -c pytorch --yes